Oct 30 21:41:47.161454 ip-10-0-133-226 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:41:47.596302 ip-10-0-133-226 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:41:47.596302 ip-10-0-133-226 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:41:47.596302 ip-10-0-133-226 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:41:47.596302 ip-10-0-133-226 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:41:47.601713 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601595    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.601713 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601712    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.601713 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601717    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601720    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601723    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601727    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601730    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601733    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601736    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601739    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601741    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601744    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601747    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601749    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601752    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601754    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601763    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601766    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601769    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601772    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601774    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601777    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.601812 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601779    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601782    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601785    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601787    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601790    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601792    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601795    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601797    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601800    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601803    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601805    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601808    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601811    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601814    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601816    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601820    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601822    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601825    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601827    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.602306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601832    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601837    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601840    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601844    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601847    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601849    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601852    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601855    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601860    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601862    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601865    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601868    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601870    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601874    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601877    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601879    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601882    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601884    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601887    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.602991 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601889    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601892    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601894    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601898    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601901    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601903    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601907    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601910    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601912    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601915    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601918    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601920    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601922    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601925    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601928    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601930    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601932    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601935    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601937    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601940    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.603544 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601942    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601945    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601949    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601953    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601956    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601958    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601961    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601964    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601966    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601969    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601972    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601974    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601976    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601979    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601981    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601984    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601986    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601989    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601992    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.604032 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601995    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.601997    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.602000    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.602002    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.602004    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.602007    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.602009    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604463    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604474    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604478    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604481    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604484    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604487    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604489    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604492    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604495    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604497    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604500    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604503    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604506    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.604499 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604509    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604512    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604515    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604517    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604520    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604523    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604525    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604528    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604530    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604532    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604535    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604537    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604539    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604542    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604545    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604549    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604551    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604554    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604558    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.605022 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604563    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604566    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604569    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604573    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604576    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604578    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604581    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604584    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604586    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604589    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604591    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604594    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604596    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604599    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604602    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604605    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604608    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604611    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604614    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604617    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.605492 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604619    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604622    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604624    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604627    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604629    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604632    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604634    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604637    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604639    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604642    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604644    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604647    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604649    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604651    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604654    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604658    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604661    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604664    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604667    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.605971 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604669    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604672    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604674    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604677    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604679    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604682    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604685    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604687    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604691    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604693    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604696    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604698    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604702    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604704    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604707    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604709    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604712    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604714    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604717    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604719    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.606440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604722    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604724    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604726    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604729    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604732    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604735    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604737    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604740    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604742    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604745    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604748    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604750    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604753    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604755    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.604758    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605310    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605317    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605320    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605323    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605326    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.606910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605330    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605333    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605336    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605339    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605343    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605345    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605348    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605351    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605353    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605356    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605358    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605361    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605363    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605366    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605368    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605371    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605373    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605376    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605378    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605381    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.607390 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605383    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605386    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605389    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605392    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605395    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605397    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605400    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605403    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605405    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605407    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605410    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605413    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605417    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605421    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605423    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605426    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605429    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605432    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605434    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.607847 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605439    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605443    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605446    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605449    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605452    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605455    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605457    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605460    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605462    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605465    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605467    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605470    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605472    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605475    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605477    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605480    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605483    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605485    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605488    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605491    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.608318 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605493    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605496    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605498    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605501    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605504    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605507    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605510    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605512    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605514    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605517    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605519    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605522    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605525    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605527    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605529    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605532    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605534    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605537    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605539    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605541    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.608780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605544    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605546    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605549    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605551    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605554    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605557    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605559    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605562    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605564    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605567    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605569    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605571    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605574    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605576    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605579    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605581    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605584    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605586    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605589    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.609301 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605592    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605594    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.605597    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606552    2576 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606563    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606572    2576 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606576    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606581    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606584    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606589    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606593    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606596    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606599    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606602    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606606    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606609    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606612    2576 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606615    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606618    2576 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606621    2576 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606626    2576 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606631    2576 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606634    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606637    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:41:47.609741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606640    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606644    2576 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606648    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606651    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606654    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606657    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606660    2576 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606664    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606666    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606670    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606672    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606676    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606680    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606682    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606687    2576 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606690    2576 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606692    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606698    2576 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606702    2576 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606705    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606708    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606711    2576 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606715    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606719    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606722    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:41:47.610471 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606725    2576 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606728    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606731    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606735    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606738    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606741    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606744    2576 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606747    2576 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606751    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606754    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606757    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606760    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606763    2576 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606766    2576 flags.go:64] FLAG: --help="false"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606769    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606773    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606776    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606778    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606782    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606785    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606788    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606791    2576 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606794    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:41:47.611093 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606797    2576 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606800    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606803    2576 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606806    2576 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606809    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606812    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606814    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606817    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606820    2576 flags.go:64] FLAG: --lock-file=""
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606823    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606826    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606829    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606836    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606838    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606841    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606844    2576 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606847    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606851    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606853    2576 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606856    2576 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606860    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606863    2576 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606867    2576 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606870    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606873    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:41:47.611649 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606876    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606881    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606884    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606886    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606890    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606897    2576 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606900    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606903    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606907    2576 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606909    2576 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606913    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606916    2576 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606918    2576 flags.go:64] FLAG: --port="10250"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606921    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606924    2576 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0856a38c16b2df750"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606928    2576 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606930    2576 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606933    2576 flags.go:64] FLAG: --register-node="true"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606936    2576 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606943    2576 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606946    2576 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606949    2576 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606952    2576 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606955    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606958    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606961    2576 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:41:47.612242 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606964    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606967    2576 flags.go:64] FLAG: --runonce="false"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606970    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606973    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606976    2576 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606979    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606982    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606987    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606990    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606993    2576 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606996    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.606999    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607002    2576 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607004    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607013    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607018    2576 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607020    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607026    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607029    2576 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607032    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607036    2576 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607039    2576 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607042    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607045    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607047    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:41:47.612888 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607052    2576 flags.go:64] FLAG: --v="2"
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607057    2576 flags.go:64] FLAG: --version="false"
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607061    2576 flags.go:64] FLAG: --vmodule=""
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607066    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607069    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607259    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607265    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607269    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607273    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607275    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607278    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607281    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607284    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607286    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607290    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607293    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607296    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607298    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607301    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607304    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.613535 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607307    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607309    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607312    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607315    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607318    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607320    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607323    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607326    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607328    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607331    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607333    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607336    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607339    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607342    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607344    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607347    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607349    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607352    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607354    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607357    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.614003 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607360    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607363    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607365    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607368    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607370    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607373    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607376    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607379    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607381    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607384    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607386    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607389    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607392    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607394    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607397    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607401    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607403    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607406    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607408    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.614598 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607411    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607413    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607416    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607419    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607421    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607425    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607428    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607430    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607433    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607436    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607439    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607441    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607444    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607447    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607449    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607452    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607455    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607457    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607460    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607465    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607467    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.615044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607470    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607472    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607475    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607478    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607480    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607483    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607485    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607488    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607491    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607493    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607496    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607498    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607501    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607504    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607506    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607509    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607512    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607515    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607517    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607520    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.615548 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607522    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.616012 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607525    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.616012 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607527    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.616012 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607530    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.616012 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607532    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.616012 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607536    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.616012 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607540    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.616012 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607542    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.616012 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607545    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.616012 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607548    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.616012 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607552    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.616012 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.607580    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:41:47.616012 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607621    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.616012 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607626    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.616012 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607629    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.616012 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607632    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607636    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607638    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607641    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607644    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607647    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607650    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607652    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607655    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607658    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607660    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607663    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607666    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607670    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607673    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607675    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607678    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607680    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607682    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607685    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.616388 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607687    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607690    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607693    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607695    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607698    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607700    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607703    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607705    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607708    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607711    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607713    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607716    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607719    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607721    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607724    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607727    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607729    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607732    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607735    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.616854 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607737    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607740    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607743    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607745    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607748    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607750    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607754    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607757    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607759    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607762    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607764    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607767    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607769    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607772    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607774    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607776    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607779    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607781    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607784    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607786    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.617324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607789    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607792    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607795    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607797    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607799    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607802    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607804    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607807    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607810    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607812    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607814    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607817    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607819    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607822    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607825    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607827    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607830    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607832    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607837    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607839    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.617791 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607842    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607844    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607847    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607849    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607852    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607854    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607857    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607860    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607864    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607867    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607870    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607873    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607875    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607881    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607884    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607888    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607892    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607895    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.618316 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607897    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607900    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607902    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607905    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607907    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.607910    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.615660    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.615676    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.616439    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.616446    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616510    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616516    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616519    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616522    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616526    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616529    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616531    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616534    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616536    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.618731 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616539    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616541    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616545    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616547    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616550    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616552    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616555    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616558    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616561    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616563    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616565    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616568    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616571    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616573    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616575    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616578    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616580    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616583    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616585    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616588    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.619184 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616591    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616595    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616598    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616601    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616603    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616606    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616608    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616611    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616614    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616616    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616620    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616624    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616627    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616630    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616633    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616635    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616638    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616640    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616643    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.619648 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616646    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616648    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616650    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616653    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616655    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616658    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616660    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616663    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616666    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616668    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616671    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616674    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616677    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616680    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616683    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616687    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616689    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616692    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616694    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616697    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.620099 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616699    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616702    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616705    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616707    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616711    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616715    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616719    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616722    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616724    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616727    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616729    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616732    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616735    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616737    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616739    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616742    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616744    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616747    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616749    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.620582 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616752    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616754    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616757    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616759    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616762    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616765    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616768    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616771    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616773    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616776    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616779    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616782    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616784    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616787    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616789    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616792    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616795    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616797    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.621044 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616800    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.621536 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.616831    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:41:47.621536 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616991    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.621536 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.616997    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.621536 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617000    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.621536 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617004    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.621536 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617007    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.621536 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617011    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.621536 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617013    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.621536 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617016    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.621536 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617019    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.621536 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617022    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.621536 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617025    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.621536 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617028    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.621536 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617031    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.621536 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617035    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617037    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617040    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617043    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617045    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617048    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617051    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617053    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617056    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617060    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617063    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617065    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617068    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617070    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617073    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617075    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617078    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617080    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617083    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617086    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.621900 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617088    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617090    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617093    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617095    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617098    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617100    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617103    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617105    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617108    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617111    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617113    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617116    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617118    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617121    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617124    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617126    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617129    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617147    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617151    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617154    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.622488 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617157    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617159    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617163    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617165    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617168    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617171    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617173    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617176    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617178    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617181    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617184    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617186    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617189    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617191    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617193    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617196    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617198    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617201    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617204    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617206    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.622956 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617209    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617212    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617215    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617217    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617220    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617222    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617225    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617227    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617230    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617233    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617235    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617238    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617241    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617243    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617245    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617248    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617251    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617254    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617256    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.623440 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617258    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.623898 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617261    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.623898 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617263    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.623898 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617266    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.623898 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617268    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.623898 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617271    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.623898 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617273    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.623898 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617276    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.623898 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617278    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.623898 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617280    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.623898 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617284    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.623898 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617288    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.623898 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617291    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.623898 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617295    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.623898 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.617319    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:41:47.623898 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617354    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617358    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617361    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617364    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617366    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617369    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617371    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617374    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617377    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617380    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617383    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617385    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617388    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617391    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617394    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617396    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617399    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617401    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617404    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617406    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:41:47.624324 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617409    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617411    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617414    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617416    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617419    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617421    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617424    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617426    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617429    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617431    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617434    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617436    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617439    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617441    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617444    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617446    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617449    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617451    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617453    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617456    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:41:47.624799 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617459    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617461    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617464    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617467    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617469    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617472    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617474    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617477    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617480    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617482    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617485    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617487    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617489    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617492    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617494    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617497    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617499    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617502    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617505    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617507    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:41:47.625317 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617510    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617512    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617515    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617517    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617520    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617523    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617525    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617528    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617530    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617533    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617535    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617538    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617540    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617543    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617545    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617547    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617550    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617552    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617555    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617557    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:41:47.625780 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617561    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617563    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617566    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617568    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617570    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617573    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617576    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617580    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617584    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617587    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617589    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617592    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617595    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617597    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617600    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617602    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617605    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617607    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617610    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:41:47.626306 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617612    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:41:47.626765 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617615    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:41:47.626765 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617617    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:41:47.626765 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617620    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:41:47.626765 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617624    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:41:47.626765 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617628    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:41:47.626765 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:47.617631    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:41:47.626765 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.617687    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:41:47.626765 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.621047    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:41:47.626765 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.622054    2576 server.go:1014] "Starting client certificate rotation"
Oct 30 21:41:47.626765 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.622174    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:41:47.626765 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.622218    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:41:47.626765 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.624474    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:41:47.629100 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.629082    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:41:47.639284 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.639264    2576 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:41:47.643487 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.643469    2576 log.go:25] "Validated CRI v1 image API"
Oct 30 21:41:47.644377 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.644355    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:41:47.648664 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.648640    2576 fs.go:135] Filesystem UUIDs: map[433e1ca6-4f16-46c9-9134-60f05ef43913:/dev/nvme0n1p4 5080a781-e22a-4fe2-9baa-9b8febaee5e0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 21:41:47.648729 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.648662    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:41:47.653384 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.653268    2576 manager.go:217] Machine: {Timestamp:2026-10-30 21:41:47.652759584 +0000 UTC m=+0.367609878 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097416 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21343e03222336fd4eea6cb9bf09ec SystemUUID:ec21343e-0322-2336-fd4e-ea6cb9bf09ec BootID:695eabdd-9568-43d9-8ae6-692fd594701d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:19:77:6b:51:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:19:77:6b:51:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:e5:4c:69:a2:7d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:41:47.653384 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.653379    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:41:47.653512 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.653500    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:41:47.656877 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.656852    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:41:47.657031 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.656878    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-226.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:41:47.657674 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.657664    2576 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:41:47.657710 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.657677    2576 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:41:47.657710 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.657691    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:41:47.658635 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.658624    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:41:47.659855 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.659844    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:41:47.664293 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.664279    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:41:47.664427 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.664418    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:41:47.665233 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.665220    2576 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:41:47.665324 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.665240    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:41:47.665324 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.665252    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:41:47.665324 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.665262    2576 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:41:47.665324 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.665272    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:41:47.665704 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.665690    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:41:47.668118 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.668100    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:41:47.674949 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.674935    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:41:47.675868 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.675856    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:41:47.675910 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.675887    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:41:47.676057 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.676037    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:41:47.676057 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.676054    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:41:47.676057 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.676060    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:41:47.676223 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.676066    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:41:47.676223 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.676072    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:41:47.676223 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.676078    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:41:47.676727 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.676718    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:41:47.676758 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.676727    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:41:47.676758 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.676735    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:41:47.676758 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.676741    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:41:47.676758 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.676750    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:41:47.676878 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.676759    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:41:47.677574 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.677564    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:41:47.677614 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.677575    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:41:47.681579 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.681563    2576 server.go:1263] "Started kubelet"
Oct 30 21:41:47.681696 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.681671    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:41:47.681753 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.681665    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:41:47.681753 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.681738    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:41:47.682549 ip-10-0-133-226 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:41:47.683597 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.683498    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:41:47.684348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.684331    2576 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:41:47.689065 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:47.689044    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:41:47.690904 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.690885    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:41:47.691515 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.691494    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:41:47.691850 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:47.691829    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:41:47.692335 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.692313    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:41:47.692335 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.692318    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:41:47.692335 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.692339    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:41:47.692520 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.692357    2576 factory.go:55] Registering systemd factory
Oct 30 21:41:47.692520 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.692406    2576 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:41:47.692520 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.692435    2576 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:41:47.692520 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.692445    2576 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:41:47.692645 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.692613    2576 factory.go:153] Registering CRI-O factory
Oct 30 21:41:47.692645 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.692638    2576 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:41:47.692696 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.692678    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:41:47.692696 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.692692    2576 factory.go:103] Registering Raw factory
Oct 30 21:41:47.692763 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.692706    2576 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:41:47.693772 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.693756    2576 manager.go:319] Starting recovery of all containers
Oct 30 21:41:47.703629 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.703609    2576 manager.go:324] Recovery completed
Oct 30 21:41:47.704882 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:47.704864    2576 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 21:41:47.707809 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.707796    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:47.709556 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.709537    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:47.709642 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.709574    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:47.709642 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.709585    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:47.710152 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.710120    2576 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:41:47.710152 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.710149    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:47.710262 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.710149    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:41:47.710262 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.710229    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:41:47.710993 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.710980    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:47.711032 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.711009    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:47.711032 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.711021    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:47.712395 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.712382    2576 policy_none.go:50] "Start"
Oct 30 21:41:47.712454 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.712399    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:41:47.712454 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.712410    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:41:47.714352 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.714341    2576 policy_none.go:44] "Start"
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.752973    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.758064    2576 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:47.758702    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.758717    2576 server.go:87] "Starting device plugin registration server"
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.758952    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.758961    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.759063    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.759193    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.759203    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:47.759728    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:47.759790    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.770515    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.771936    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.771970    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.772013    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:41:47.775717 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:47.772048    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 21:41:47.821847 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.821812    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-226.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:41:47.859609 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.859573    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:47.861296 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.861274    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:47.861399 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.861312    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:47.861399 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.861323    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:47.861399 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.861352    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:47.869782 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:47.869737    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-226.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 21:41:47.870651 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:47.869735    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-226.us-west-1.compute.internal.18e36c9f35c89e5f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-226.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-226.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:41:47.681529439 +0000 UTC m=+0.396379732,LastTimestamp:2026-10-30 21:41:47.681529439 +0000 UTC m=+0.396379732,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-226.us-west-1.compute.internal,}"
Oct 30 21:41:47.872171 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.872152    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"]
Oct 30 21:41:47.872255 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.872219    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:47.873048 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.873029    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:47.873153 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.873063    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:47.873153 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.873090    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:47.875343 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.875329    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:47.875526 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.875511    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:47.875564 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.875542    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:47.876076 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.876058    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:47.876191 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.876083    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:47.876191 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.876098    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:47.876191 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.876108    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:47.876191 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.876126    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:47.876191 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.876158    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:47.878277 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.878254    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:47.878358 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.878281    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:47.878967 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.878953    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:47.879038 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.878978    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:47.879038 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.878987    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:47.893648 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.893628    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c0e7e76dbf05397217f3bc5792d5aaba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"c0e7e76dbf05397217f3bc5792d5aaba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:47.893726 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.893652    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0e7e76dbf05397217f3bc5792d5aaba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"c0e7e76dbf05397217f3bc5792d5aaba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:47.893726 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.893672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/24d6104f1ec87a54e6dc2f912238418d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"24d6104f1ec87a54e6dc2f912238418d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:47.903764 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:47.903747    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found" node="ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:47.906627 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:47.906611    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found" node="ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:47.918303 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.918272    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:41:47.918408 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.918344    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qvbfm"
Oct 30 21:41:47.925567 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.925549    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qvbfm"
Oct 30 21:41:47.976396 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:47.976341    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 21:41:47.976396 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:47.976340    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-226.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 21:41:47.993842 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.993797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c0e7e76dbf05397217f3bc5792d5aaba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"c0e7e76dbf05397217f3bc5792d5aaba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:47.993842 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.993845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0e7e76dbf05397217f3bc5792d5aaba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"c0e7e76dbf05397217f3bc5792d5aaba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:47.994013 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.993864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/24d6104f1ec87a54e6dc2f912238418d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"24d6104f1ec87a54e6dc2f912238418d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:47.994013 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.993889    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c0e7e76dbf05397217f3bc5792d5aaba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"c0e7e76dbf05397217f3bc5792d5aaba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:47.994013 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.993900    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/24d6104f1ec87a54e6dc2f912238418d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"24d6104f1ec87a54e6dc2f912238418d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:47.994013 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:47.993933    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0e7e76dbf05397217f3bc5792d5aaba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"c0e7e76dbf05397217f3bc5792d5aaba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:48.025199 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:48.025170    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 21:41:48.025329 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:48.025107    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-226.us-west-1.compute.internal.18e36c9f3774452e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-226.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-226.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-226.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:41:47.709556014 +0000 UTC m=+0.424406307,LastTimestamp:2026-10-30 21:41:47.709556014 +0000 UTC m=+0.424406307,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-226.us-west-1.compute.internal,}"
Oct 30 21:41:48.028304 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.028281    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:48.028428 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:48.028356    2576 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-133-226.us-west-1.compute.internal\" not found" node="ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:48.175599 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:48.175518    2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-226.us-west-1.compute.internal\" not found" node="ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:48.208091 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.208062    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:48.212106 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.212075    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:48.229248 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.229227    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:48.230072 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.230052    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:48.230189 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.230086    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:48.230189 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.230097    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:48.230189 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.230128    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:48.287538 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.287513    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:48.287669 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:48.287542    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-226.us-west-1.compute.internal\": node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:41:48.400577 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:48.400547    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:41:48.501586 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:48.501490    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:41:48.602158 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:48.602102    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:41:48.622312 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.622281    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:41:48.622497 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.622475    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:41:48.622615 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:48.622529    2576 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a3ee7570faedf4a5ca3084ec2df71db2-8c14c679226562c9.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-226.us-west-1.compute.internal.18e36c9f3774c733\": read tcp 10.0.133.226:47152->77.112.223.82:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-226.us-west-1.compute.internal.18e36c9f3774c733  default   5157 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-226.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-226.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-226.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:41:47 +0000 UTC,LastTimestamp:2026-10-30 21:41:47.861327064 +0000 UTC m=+0.576177358,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-226.us-west-1.compute.internal,}"
Oct 30 21:41:48.703271 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:48.703239    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:41:48.803824 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:48.803739    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:41:48.904245 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:48.904205    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:41:48.931728 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.931683    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:36:48 +0000 UTC" deadline="2028-06-22 10:23:10.006941328 +0000 UTC"
Oct 30 21:41:48.931728 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.931724    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14412h41m21.075220848s"
Oct 30 21:41:48.994484 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.994448    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:41:48.995445 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.995427    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:41:48.995557 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.995465    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:41:48.995557 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:48.995480    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:41:49.004732 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:49.004710    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:41:49.105803 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:49.105772    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:41:49.128318 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.128292    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:49.206957 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:49.206920    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:41:49.308103 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:49.308069    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:41:49.311699 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:49.311642    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d6104f1ec87a54e6dc2f912238418d.slice/crio-d5e9e682eb747437d890f42d5d370b3d3251267f18458346a398f78419834eae WatchSource:0}: Error finding container d5e9e682eb747437d890f42d5d370b3d3251267f18458346a398f78419834eae: Status 404 returned error can't find the container with id d5e9e682eb747437d890f42d5d370b3d3251267f18458346a398f78419834eae
Oct 30 21:41:49.313164 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:49.313139    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e7e76dbf05397217f3bc5792d5aaba.slice/crio-414b472a25b2ce5a0df7e80951a1b5acf023d6c1c217e166ffc9ed3657aabec6 WatchSource:0}: Error finding container 414b472a25b2ce5a0df7e80951a1b5acf023d6c1c217e166ffc9ed3657aabec6: Status 404 returned error can't find the container with id 414b472a25b2ce5a0df7e80951a1b5acf023d6c1c217e166ffc9ed3657aabec6
Oct 30 21:41:49.317777 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.317759    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:41:49.408685 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:49.408582    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:41:49.480036 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.480012    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:49.509765 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:49.509736    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:41:49.576858 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.576825    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:49.592666 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.592636    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:49.658531 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.658497    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:41:49.659785 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.659722    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:41:49.666190 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.666174    2576 apiserver.go:52] "Watching apiserver"
Oct 30 21:41:49.693457 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.693417    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:41:49.722602 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.722573    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:41:49.777054 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.777009    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:41:49.777448 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.777420    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-9hg74","openshift-multus/network-metrics-daemon-8kpvh","kube-system/global-pull-secret-syncer-x7hcg","kube-system/konnectivity-agent-prfrp","openshift-image-registry/node-ca-rr82t","openshift-multus/multus-9g8ct","openshift-network-operator/iptables-alerter-4wg8g","openshift-ovn-kubernetes/ovnkube-node-s8pzx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw","openshift-cluster-node-tuning-operator/tuned-ch6rj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-l25db","openshift-network-diagnostics/network-check-target-7d5lt","kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"]
Oct 30 21:41:49.781394 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.781370    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:41:49.783748 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.783729    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:41:49.783854 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:49.783813    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7d5lt" podUID="946b7964-c272-43fc-84f3-2e37be76568f"
Oct 30 21:41:49.785972 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.785954    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:49.786073 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:49.786029    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7hcg" podUID="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369"
Oct 30 21:41:49.788170 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.788150    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:41:49.790630 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.790611    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:41:49.793196 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.793178    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.795666 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.795645    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:49.797981 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.797963    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:41:49.800419 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.800399    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:41:49.800492 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:49.800475    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpvh" podUID="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83"
Oct 30 21:41:49.802864 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.802846    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:49.804299 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-dbus\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:49.804402 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-cni-multus\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.804402 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804324    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f970ab0-3a72-4464-b767-d6867503e81d-host-slash\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:41:49.804402 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804342    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-kubelet-config\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:49.804402 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804358    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-socket-dir-parent\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.804617 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804419    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04aee07d-cc70-47b8-b101-d2b3b7111198-serviceca\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:41:49.804617 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804451    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcrtw\" (UniqueName: \"kubernetes.io/projected/04aee07d-cc70-47b8-b101-d2b3b7111198-kube-api-access-bcrtw\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:41:49.804617 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804468    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-cni-binary-copy\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.804617 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804484    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88edf321-028a-4bb5-b729-1359bd54fb57-agent-certs\") pod \"konnectivity-agent-prfrp\" (UID: \"88edf321-028a-4bb5-b729-1359bd54fb57\") " pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:41:49.804617 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804516    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04aee07d-cc70-47b8-b101-d2b3b7111198-host\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:41:49.804617 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-cni-bin\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.804617 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804558    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-conf-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.804617 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804610    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdm95\" (UniqueName: \"kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95\") pod \"network-check-target-7d5lt\" (UID: \"946b7964-c272-43fc-84f3-2e37be76568f\") " pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:41:49.805035 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804632    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88edf321-028a-4bb5-b729-1359bd54fb57-konnectivity-ca\") pod \"konnectivity-agent-prfrp\" (UID: \"88edf321-028a-4bb5-b729-1359bd54fb57\") " pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:41:49.805035 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-cnibin\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.805035 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804703    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-netns\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.805035 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804728    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-hostroot\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.805035 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804758    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-etc-kubernetes\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.805035 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804789    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfh9d\" (UniqueName: \"kubernetes.io/projected/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-kube-api-access-jfh9d\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.805035 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804822    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:49.805035 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804850    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-cni-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.805035 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-kubelet\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.805035 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-multus-certs\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.805035 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804945    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-system-cni-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.805035 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804972    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-daemon-config\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.805035 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.804991    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f970ab0-3a72-4464-b767-d6867503e81d-iptables-alerter-script\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:41:49.805035 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.805029    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z876\" (UniqueName: \"kubernetes.io/projected/4f970ab0-3a72-4464-b767-d6867503e81d-kube-api-access-7z876\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:41:49.805760 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.805054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-k8s-cni-cncf-io\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.805760 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.805415    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.807695 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.807653    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal" event={"ID":"24d6104f1ec87a54e6dc2f912238418d","Type":"ContainerStarted","Data":"d5e9e682eb747437d890f42d5d370b3d3251267f18458346a398f78419834eae"}
Oct 30 21:41:49.807796 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.807701    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal" event={"ID":"c0e7e76dbf05397217f3bc5792d5aaba","Type":"ContainerStarted","Data":"414b472a25b2ce5a0df7e80951a1b5acf023d6c1c217e166ffc9ed3657aabec6"}
Oct 30 21:41:49.807796 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.807755    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:49.841917 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.841880    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:41:49.842479 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.842421    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:41:49.842479 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.842447    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:41:49.842723 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.842706    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pkpl9\""
Oct 30 21:41:49.843520 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.843502    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-msnq6\""
Oct 30 21:41:49.855512 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.855490    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-82rrg\""
Oct 30 21:41:49.855625 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.855534    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:41:49.855625 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.855546    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:41:49.856073 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.856052    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p7xlk\""
Oct 30 21:41:49.856140 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.856086    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:41:49.856213 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.856180    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:41:49.856213 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.856186    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:41:49.856213 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.856195    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:41:49.856342 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.856236    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:41:49.856342 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.856258    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:41:49.856342 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.856268    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:41:49.856342 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.856268    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:41:49.856342 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.856327    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:41:49.885899 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.885838    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:41:49.886245 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.886219    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:41:49.886339 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.886250    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:41:49.886339 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.886291    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:41:49.886546 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.886396    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:41:49.886546 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.886454    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:41:49.886546 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.886476    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:41:49.886546 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.886477    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:41:49.886546 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.886479    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:41:49.888807 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.888779    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:41:49.888912 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.888817    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:41:49.893975 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.893945    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:41:49.895679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.895648    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:41:49.895679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.895667    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6gscd\""
Oct 30 21:41:49.895679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.895679    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vtv5j\""
Oct 30 21:41:49.895918 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.895701    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:41:49.895918 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.895713    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:41:49.895918 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.895725    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7tj7w\""
Oct 30 21:41:49.895918 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.895765    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9qhlv\""
Oct 30 21:41:49.895918 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.895788    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-phmwh"
Oct 30 21:41:49.895918 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.895843    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:41:49.896162 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.895937    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zffrj\""
Oct 30 21:41:49.906102 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906057    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-socket-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:49.906102 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906100    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-host\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:49.906301 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906117    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-systemd\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.906301 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906215    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdm95\" (UniqueName: \"kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95\") pod \"network-check-target-7d5lt\" (UID: \"946b7964-c272-43fc-84f3-2e37be76568f\") " pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:41:49.906301 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906281    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88edf321-028a-4bb5-b729-1359bd54fb57-konnectivity-ca\") pod \"konnectivity-agent-prfrp\" (UID: \"88edf321-028a-4bb5-b729-1359bd54fb57\") " pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:41:49.906437 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-cni-netd\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.906490 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-env-overrides\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.906546 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906505    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:41:49.906598 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906549    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-cni-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.906598 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906583    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-kubelet\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.906696 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906612    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-kubernetes\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:49.906696 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysctl-d\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:49.906696 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-run\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:49.906830 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-log-socket\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.906830 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906749    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-kubelet\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.906830 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906775    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.906830 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906799    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-cni-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.906830 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906808    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cni-binary-copy\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:49.907003 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906849    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f970ab0-3a72-4464-b767-d6867503e81d-iptables-alerter-script\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:41:49.907003 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906875    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z876\" (UniqueName: \"kubernetes.io/projected/4f970ab0-3a72-4464-b767-d6867503e81d-kube-api-access-7z876\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:41:49.907003 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906895    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-tuned\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:49.907003 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906912    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-ovn\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.907003 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906943    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovnkube-script-lib\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.907003 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906960    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88edf321-028a-4bb5-b729-1359bd54fb57-konnectivity-ca\") pod \"konnectivity-agent-prfrp\" (UID: \"88edf321-028a-4bb5-b729-1359bd54fb57\") " pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:41:49.907328 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.906968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:49.907328 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-k8s-cni-cncf-io\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.907328 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:49.907328 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f970ab0-3a72-4464-b767-d6867503e81d-host-slash\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:41:49.907328 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907097    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-k8s-cni-cncf-io\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.907328 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-modprobe-d\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:49.907328 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907176    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f970ab0-3a72-4464-b767-d6867503e81d-host-slash\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:41:49.907328 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907195    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:49.907328 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zvtp\" (UniqueName: \"kubernetes.io/projected/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-kube-api-access-4zvtp\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:49.907328 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-socket-dir-parent\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907340    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f970ab0-3a72-4464-b767-d6867503e81d-iptables-alerter-script\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907337    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907351    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-socket-dir-parent\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907388    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04aee07d-cc70-47b8-b101-d2b3b7111198-serviceca\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcrtw\" (UniqueName: \"kubernetes.io/projected/04aee07d-cc70-47b8-b101-d2b3b7111198-kube-api-access-bcrtw\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907468    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-cni-binary-copy\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-kubelet\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907506    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovnkube-config\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907522    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvmtp\" (UniqueName: \"kubernetes.io/projected/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-kube-api-access-hvmtp\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-sys-fs\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88edf321-028a-4bb5-b729-1359bd54fb57-agent-certs\") pod \"konnectivity-agent-prfrp\" (UID: \"88edf321-028a-4bb5-b729-1359bd54fb57\") " pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-cni-bin\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-conf-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07039a3-5a8c-4bdf-8241-47942c8d359d-tmp\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907678    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-cni-bin\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907695    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-node-log\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.907833 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907721    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-var-lib-kubelet\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907726    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-conf-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907765    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-cnibin\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907799    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-cnibin\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907800    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-netns\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-hostroot\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907844    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-netns\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-etc-kubernetes\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907875    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfh9d\" (UniqueName: \"kubernetes.io/projected/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-kube-api-access-jfh9d\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907871    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04aee07d-cc70-47b8-b101-d2b3b7111198-serviceca\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907915    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-etc-kubernetes\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907916    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907964    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9430e64-a396-496f-a23a-96439631e58a-hosts-file\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.907902    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-hostroot\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908126    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-cni-binary-copy\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908095    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghwzq\" (UniqueName: \"kubernetes.io/projected/f9430e64-a396-496f-a23a-96439631e58a-kube-api-access-ghwzq\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kr6b\" (UniqueName: \"kubernetes.io/projected/f07039a3-5a8c-4bdf-8241-47942c8d359d-kube-api-access-8kr6b\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:49.908679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908259    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-multus-certs\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908315    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-system-cni-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908328    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-multus-certs\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908343    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-daemon-config\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-sys\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908385    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-system-cni-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908396    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-systemd-units\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:49.908373    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbfft\" (UniqueName: \"kubernetes.io/projected/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-kube-api-access-sbfft\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908454    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npb5n\" (UniqueName: \"kubernetes.io/projected/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-kube-api-access-npb5n\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:49.908481    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret podName:4d1fccd5-6eaa-4a43-9a1e-d67c1005e369 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:50.408457494 +0000 UTC m=+3.123307816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret") pod "global-pull-secret-syncer-x7hcg" (UID: "4d1fccd5-6eaa-4a43-9a1e-d67c1005e369") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908519    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-lib-modules\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908551    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-run-netns\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908577    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-etc-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908605    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovn-node-metrics-cert\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.909348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908656    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-dbus\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-cni-multus\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908736    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-dbus\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908771    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-daemon-config\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908785    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-registration-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908796    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-cni-multus\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-kubelet-config\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908853    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-slash\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908872    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-var-lib-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-system-cni-dir\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908918    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-device-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9430e64-a396-496f-a23a-96439631e58a-tmp-dir\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908934    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-kubelet-config\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysctl-conf\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.908994    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-systemd\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.909023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.909047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04aee07d-cc70-47b8-b101-d2b3b7111198-host\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:41:49.909870 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.909063    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysconfig\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:49.910404 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.909072    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04aee07d-cc70-47b8-b101-d2b3b7111198-host\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:41:49.910404 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.909084    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.910404 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.909102    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-cni-bin\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:49.910404 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.909118    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cnibin\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:49.911761 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.911738    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88edf321-028a-4bb5-b729-1359bd54fb57-agent-certs\") pod \"konnectivity-agent-prfrp\" (UID: \"88edf321-028a-4bb5-b729-1359bd54fb57\") " pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:41:49.966285 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:49.966253    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:41:49.966285 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:49.966283    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:41:49.966285 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:49.966295    2576 projected.go:196] Error preparing data for projected volume kube-api-access-sdm95 for pod openshift-network-diagnostics/network-check-target-7d5lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:49.966518 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:49.966364    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95 podName:946b7964-c272-43fc-84f3-2e37be76568f nodeName:}" failed. No retries permitted until 2026-10-30 21:41:50.46634569 +0000 UTC m=+3.181195972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sdm95" (UniqueName: "kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95") pod "network-check-target-7d5lt" (UID: "946b7964-c272-43fc-84f3-2e37be76568f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:49.968283 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.968254    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z876\" (UniqueName: \"kubernetes.io/projected/4f970ab0-3a72-4464-b767-d6867503e81d-kube-api-access-7z876\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:41:49.969267 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.969248    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcrtw\" (UniqueName: \"kubernetes.io/projected/04aee07d-cc70-47b8-b101-d2b3b7111198-kube-api-access-bcrtw\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:41:49.969438 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:49.969421    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfh9d\" (UniqueName: \"kubernetes.io/projected/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-kube-api-access-jfh9d\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:50.009780 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.009742    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.009780 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.009787    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cni-binary-copy\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:50.010010 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.009806    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-tuned\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.010010 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.009849    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.010010 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.009910    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-ovn\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.010010 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.009948    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovnkube-script-lib\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.010010 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.009975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:50.010010 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:50.010010 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010004    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-ovn\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.010408 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010030    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-modprobe-d\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.010408 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010124    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:50.010408 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010179    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-modprobe-d\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.010408 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010180    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zvtp\" (UniqueName: \"kubernetes.io/projected/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-kube-api-access-4zvtp\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:50.010408 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010184    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:50.010408 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010201    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:50.010408 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010237    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:50.010408 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010294    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-kubelet\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.010408 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010343    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-kubelet\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.010408 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010376    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovnkube-config\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.010408 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmtp\" (UniqueName: \"kubernetes.io/projected/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-kube-api-access-hvmtp\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010430    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-sys-fs\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07039a3-5a8c-4bdf-8241-47942c8d359d-tmp\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010496    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-sys-fs\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010490    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cni-binary-copy\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-node-log\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010559    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-node-log\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-var-lib-kubelet\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010648    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9430e64-a396-496f-a23a-96439631e58a-hosts-file\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010650    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-var-lib-kubelet\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010678    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghwzq\" (UniqueName: \"kubernetes.io/projected/f9430e64-a396-496f-a23a-96439631e58a-kube-api-access-ghwzq\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010701    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kr6b\" (UniqueName: \"kubernetes.io/projected/f07039a3-5a8c-4bdf-8241-47942c8d359d-kube-api-access-8kr6b\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010718    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9430e64-a396-496f-a23a-96439631e58a-hosts-file\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-sys\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010784    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-systemd-units\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010792    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovnkube-script-lib\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.010966 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfft\" (UniqueName: \"kubernetes.io/projected/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-kube-api-access-sbfft\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npb5n\" (UniqueName: \"kubernetes.io/projected/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-kube-api-access-npb5n\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010858    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-sys\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010869    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-lib-modules\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010893    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-run-netns\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-etc-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010805    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovn-node-metrics-cert\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010988    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-registration-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010998    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovnkube-config\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-slash\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011029    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-etc-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011054    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-var-lib-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.010987    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-systemd-units\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011035    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-run-netns\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011089    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-lib-modules\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011092    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-system-cni-dir\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:50.011850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011128    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-slash\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011133    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-system-cni-dir\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011139    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-var-lib-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011095    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-registration-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-device-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011281    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9430e64-a396-496f-a23a-96439631e58a-tmp-dir\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011322    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysctl-conf\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-systemd\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysconfig\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011452    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-systemd\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011484    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-cni-bin\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011512    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cnibin\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-socket-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-host\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011569    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:50.012534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011590    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-systemd\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011598    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysctl-conf\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011625    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9430e64-a396-496f-a23a-96439631e58a-tmp-dir\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011641    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysconfig\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011588    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011667    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-host\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-cni-netd\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011695    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cnibin\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011702    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-cni-bin\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-env-overrides\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011742    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-socket-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011739    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-systemd\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011765    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011774    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-cni-netd\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-kubernetes\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011736    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysctl-d\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011860    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-kubernetes\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.013105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011874    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-run\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.013711 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:50.011938    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:50.013711 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.011959    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-device-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:50.013711 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:50.012006    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs podName:d2f6dd07-b2ef-43b9-af3a-9bf016a45c83 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:50.51198767 +0000 UTC m=+3.226837970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs") pod "network-metrics-daemon-8kpvh" (UID: "d2f6dd07-b2ef-43b9-af3a-9bf016a45c83") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:50.013711 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.012020    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-run\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.013711 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.012027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-log-socket\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.013711 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.012082    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysctl-d\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.013711 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.012098    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-log-socket\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.013711 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.012126    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-env-overrides\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.013711 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.012257    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:50.013711 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.013144    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-tuned\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.013711 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.013535    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07039a3-5a8c-4bdf-8241-47942c8d359d-tmp\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.014350 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.014330    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovn-node-metrics-cert\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.073033 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.072998    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kr6b\" (UniqueName: \"kubernetes.io/projected/f07039a3-5a8c-4bdf-8241-47942c8d359d-kube-api-access-8kr6b\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.073238 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.073000    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvmtp\" (UniqueName: \"kubernetes.io/projected/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-kube-api-access-hvmtp\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.073321 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.073243    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbfft\" (UniqueName: \"kubernetes.io/projected/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-kube-api-access-sbfft\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:50.073321 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.073307    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghwzq\" (UniqueName: \"kubernetes.io/projected/f9430e64-a396-496f-a23a-96439631e58a-kube-api-access-ghwzq\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:41:50.073405 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.073328    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zvtp\" (UniqueName: \"kubernetes.io/projected/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-kube-api-access-4zvtp\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:50.073405 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.073378    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npb5n\" (UniqueName: \"kubernetes.io/projected/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-kube-api-access-npb5n\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:41:50.093337 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.093310    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:41:50.102429 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.102402    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:41:50.104458 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:50.104429    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f970ab0_3a72_4464_b767_d6867503e81d.slice/crio-3e0c5f28917c5e12df3379aee4c16573ddaa61395e446cd45321b8d79415d73f WatchSource:0}: Error finding container 3e0c5f28917c5e12df3379aee4c16573ddaa61395e446cd45321b8d79415d73f: Status 404 returned error can't find the container with id 3e0c5f28917c5e12df3379aee4c16573ddaa61395e446cd45321b8d79415d73f
Oct 30 21:41:50.109393 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.109373    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:41:50.111910 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:50.111883    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88edf321_028a_4bb5_b729_1359bd54fb57.slice/crio-34b9ef2673106cb292bf45d813a921de92268e9d16f98a1f6a51edb850bd94a1 WatchSource:0}: Error finding container 34b9ef2673106cb292bf45d813a921de92268e9d16f98a1f6a51edb850bd94a1: Status 404 returned error can't find the container with id 34b9ef2673106cb292bf45d813a921de92268e9d16f98a1f6a51edb850bd94a1
Oct 30 21:41:50.113617 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.113604    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9g8ct"
Oct 30 21:41:50.119471 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:50.119448    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04aee07d_cc70_47b8_b101_d2b3b7111198.slice/crio-3baa19f3af396a1f221a972f4506f42e206e135a6cb1a8d09dd35e822303be3a WatchSource:0}: Error finding container 3baa19f3af396a1f221a972f4506f42e206e135a6cb1a8d09dd35e822303be3a: Status 404 returned error can't find the container with id 3baa19f3af396a1f221a972f4506f42e206e135a6cb1a8d09dd35e822303be3a
Oct 30 21:41:50.124685 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.124670    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:41:50.124993 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:50.124973    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd5141f_18f5_4f4d_a4d2_5f1e170a68e2.slice/crio-e316df61719a014965aaec92d5c4c1b1730a803c5dd9d96dafedd8d68ef3e52f WatchSource:0}: Error finding container e316df61719a014965aaec92d5c4c1b1730a803c5dd9d96dafedd8d68ef3e52f: Status 404 returned error can't find the container with id e316df61719a014965aaec92d5c4c1b1730a803c5dd9d96dafedd8d68ef3e52f
Oct 30 21:41:50.125694 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.125678    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:41:50.131899 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.131875    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:41:50.136509 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:50.136479    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0fbfbe_62ce_4e9c_96b9_066d7bf1bfd3.slice/crio-b4738fd074784866f876219762ab832e8877702a595da98471afaf3b97968b5e WatchSource:0}: Error finding container b4738fd074784866f876219762ab832e8877702a595da98471afaf3b97968b5e: Status 404 returned error can't find the container with id b4738fd074784866f876219762ab832e8877702a595da98471afaf3b97968b5e
Oct 30 21:41:50.139688 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:50.139656    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9430e64_a396_496f_a23a_96439631e58a.slice/crio-ca1e93b98daa51c4a97fba771bf8881d8c845b6006db5a50f711c6a952c067ed WatchSource:0}: Error finding container ca1e93b98daa51c4a97fba771bf8881d8c845b6006db5a50f711c6a952c067ed: Status 404 returned error can't find the container with id ca1e93b98daa51c4a97fba771bf8881d8c845b6006db5a50f711c6a952c067ed
Oct 30 21:41:50.143619 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:50.143497    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07039a3_5a8c_4bdf_8241_47942c8d359d.slice/crio-5b97e8f0591b06cdd1bc7711811e74e61fee2733392a43d8091d50e5873a3129 WatchSource:0}: Error finding container 5b97e8f0591b06cdd1bc7711811e74e61fee2733392a43d8091d50e5873a3129: Status 404 returned error can't find the container with id 5b97e8f0591b06cdd1bc7711811e74e61fee2733392a43d8091d50e5873a3129
Oct 30 21:41:50.161755 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.161670    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:41:50.168071 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.168037    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:41:50.176567 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:50.176537    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0ce829_dc72_4f8e_91ea_2a5313fa7b11.slice/crio-9e203ac76cdf33263fb80e851d1d322a52020e930e8f789f1078c9aa90a1601a WatchSource:0}: Error finding container 9e203ac76cdf33263fb80e851d1d322a52020e930e8f789f1078c9aa90a1601a: Status 404 returned error can't find the container with id 9e203ac76cdf33263fb80e851d1d322a52020e930e8f789f1078c9aa90a1601a
Oct 30 21:41:50.184653 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:41:50.184617    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb9603c_7aa1_4d59_afc7_5fc921bf2505.slice/crio-9390004a5986cc084c3803f6262f3703bea203c3b145ac3be46ea854f4128d9f WatchSource:0}: Error finding container 9390004a5986cc084c3803f6262f3703bea203c3b145ac3be46ea854f4128d9f: Status 404 returned error can't find the container with id 9390004a5986cc084c3803f6262f3703bea203c3b145ac3be46ea854f4128d9f
Oct 30 21:41:50.415880 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.415766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:50.416065 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:50.415972    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:50.416065 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:50.416037    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret podName:4d1fccd5-6eaa-4a43-9a1e-d67c1005e369 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:51.416020954 +0000 UTC m=+4.130871236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret") pod "global-pull-secret-syncer-x7hcg" (UID: "4d1fccd5-6eaa-4a43-9a1e-d67c1005e369") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:50.516552 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.516510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdm95\" (UniqueName: \"kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95\") pod \"network-check-target-7d5lt\" (UID: \"946b7964-c272-43fc-84f3-2e37be76568f\") " pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:41:50.516712 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.516566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:41:50.516763 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:50.516747    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:50.516832 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:50.516821    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs podName:d2f6dd07-b2ef-43b9-af3a-9bf016a45c83 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:51.51680132 +0000 UTC m=+4.231651618 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs") pod "network-metrics-daemon-8kpvh" (UID: "d2f6dd07-b2ef-43b9-af3a-9bf016a45c83") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:50.516947 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:50.516923    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:41:50.516947 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:50.516950    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:41:50.517164 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:50.516963    2576 projected.go:196] Error preparing data for projected volume kube-api-access-sdm95 for pod openshift-network-diagnostics/network-check-target-7d5lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:50.517164 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:50.517001    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95 podName:946b7964-c272-43fc-84f3-2e37be76568f nodeName:}" failed. No retries permitted until 2026-10-30 21:41:51.516988567 +0000 UTC m=+4.231838877 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdm95" (UniqueName: "kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95") pod "network-check-target-7d5lt" (UID: "946b7964-c272-43fc-84f3-2e37be76568f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:50.783330 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.783227    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9hg74" event={"ID":"f9430e64-a396-496f-a23a-96439631e58a","Type":"ContainerStarted","Data":"ca1e93b98daa51c4a97fba771bf8881d8c845b6006db5a50f711c6a952c067ed"}
Oct 30 21:41:50.785595 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.785562    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9g8ct" event={"ID":"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2","Type":"ContainerStarted","Data":"e316df61719a014965aaec92d5c4c1b1730a803c5dd9d96dafedd8d68ef3e52f"}
Oct 30 21:41:50.786957 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.786924    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rr82t" event={"ID":"04aee07d-cc70-47b8-b101-d2b3b7111198","Type":"ContainerStarted","Data":"3baa19f3af396a1f221a972f4506f42e206e135a6cb1a8d09dd35e822303be3a"}
Oct 30 21:41:50.788468 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.788399    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l25db" event={"ID":"6eb9603c-7aa1-4d59-afc7-5fc921bf2505","Type":"ContainerStarted","Data":"9390004a5986cc084c3803f6262f3703bea203c3b145ac3be46ea854f4128d9f"}
Oct 30 21:41:50.789984 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.789739    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw" event={"ID":"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3","Type":"ContainerStarted","Data":"b4738fd074784866f876219762ab832e8877702a595da98471afaf3b97968b5e"}
Oct 30 21:41:50.791105 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.791050    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-prfrp" event={"ID":"88edf321-028a-4bb5-b729-1359bd54fb57","Type":"ContainerStarted","Data":"34b9ef2673106cb292bf45d813a921de92268e9d16f98a1f6a51edb850bd94a1"}
Oct 30 21:41:50.792530 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.792501    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4wg8g" event={"ID":"4f970ab0-3a72-4464-b767-d6867503e81d","Type":"ContainerStarted","Data":"3e0c5f28917c5e12df3379aee4c16573ddaa61395e446cd45321b8d79415d73f"}
Oct 30 21:41:50.794177 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.794131    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx" event={"ID":"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11","Type":"ContainerStarted","Data":"9e203ac76cdf33263fb80e851d1d322a52020e930e8f789f1078c9aa90a1601a"}
Oct 30 21:41:50.795515 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.795491    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ch6rj" event={"ID":"f07039a3-5a8c-4bdf-8241-47942c8d359d","Type":"ContainerStarted","Data":"5b97e8f0591b06cdd1bc7711811e74e61fee2733392a43d8091d50e5873a3129"}
Oct 30 21:41:50.899334 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.899266    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:50 +0000 UTC" deadline="2028-08-01 07:17:50.294263992 +0000 UTC"
Oct 30 21:41:50.899334 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:50.899329    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15369h35m59.394940085s"
Oct 30 21:41:51.426642 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:51.426597    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:51.426858 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:51.426839    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:51.426948 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:51.426912    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret podName:4d1fccd5-6eaa-4a43-9a1e-d67c1005e369 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:53.426891347 +0000 UTC m=+6.141741650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret") pod "global-pull-secret-syncer-x7hcg" (UID: "4d1fccd5-6eaa-4a43-9a1e-d67c1005e369") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:51.527837 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:51.527796    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdm95\" (UniqueName: \"kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95\") pod \"network-check-target-7d5lt\" (UID: \"946b7964-c272-43fc-84f3-2e37be76568f\") " pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:41:51.528013 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:51.527853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:41:51.528093 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:51.528015    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:51.528093 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:51.528087    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs podName:d2f6dd07-b2ef-43b9-af3a-9bf016a45c83 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:53.528066541 +0000 UTC m=+6.242916838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs") pod "network-metrics-daemon-8kpvh" (UID: "d2f6dd07-b2ef-43b9-af3a-9bf016a45c83") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:51.528235 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:51.528175    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:41:51.528235 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:51.528210    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:41:51.528235 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:51.528225    2576 projected.go:196] Error preparing data for projected volume kube-api-access-sdm95 for pod openshift-network-diagnostics/network-check-target-7d5lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:51.528382 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:51.528267    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95 podName:946b7964-c272-43fc-84f3-2e37be76568f nodeName:}" failed. No retries permitted until 2026-10-30 21:41:53.528252094 +0000 UTC m=+6.243102392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdm95" (UniqueName: "kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95") pod "network-check-target-7d5lt" (UID: "946b7964-c272-43fc-84f3-2e37be76568f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:51.774860 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:51.774373    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:51.774860 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:51.774414    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:41:51.774860 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:51.774535    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7hcg" podUID="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369"
Oct 30 21:41:51.774860 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:51.774621    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpvh" podUID="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83"
Oct 30 21:41:51.774860 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:51.774728    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:41:51.774860 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:51.774825    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7d5lt" podUID="946b7964-c272-43fc-84f3-2e37be76568f"
Oct 30 21:41:51.900428 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:51.900352    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:50 +0000 UTC" deadline="2028-05-24 18:07:22.061191775 +0000 UTC"
Oct 30 21:41:51.900428 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:51.900395    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13724h25m30.160801796s"
Oct 30 21:41:53.446991 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:53.446713    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:53.447595 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:53.447043    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:53.447595 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:53.447136    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret podName:4d1fccd5-6eaa-4a43-9a1e-d67c1005e369 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:57.447116137 +0000 UTC m=+10.161966435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret") pod "global-pull-secret-syncer-x7hcg" (UID: "4d1fccd5-6eaa-4a43-9a1e-d67c1005e369") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:53.548130 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:53.548085    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdm95\" (UniqueName: \"kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95\") pod \"network-check-target-7d5lt\" (UID: \"946b7964-c272-43fc-84f3-2e37be76568f\") " pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:41:53.548364 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:53.548146    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:41:53.548482 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:53.548367    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:53.548482 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:53.548447    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs podName:d2f6dd07-b2ef-43b9-af3a-9bf016a45c83 nodeName:}" failed. No retries permitted until 2026-10-30 21:41:57.548425624 +0000 UTC m=+10.263275924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs") pod "network-metrics-daemon-8kpvh" (UID: "d2f6dd07-b2ef-43b9-af3a-9bf016a45c83") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:53.548604 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:53.548557    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:41:53.548604 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:53.548573    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:41:53.548604 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:53.548585    2576 projected.go:196] Error preparing data for projected volume kube-api-access-sdm95 for pod openshift-network-diagnostics/network-check-target-7d5lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:53.548748 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:53.548627    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95 podName:946b7964-c272-43fc-84f3-2e37be76568f nodeName:}" failed. No retries permitted until 2026-10-30 21:41:57.548613351 +0000 UTC m=+10.263463649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdm95" (UniqueName: "kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95") pod "network-check-target-7d5lt" (UID: "946b7964-c272-43fc-84f3-2e37be76568f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:53.774598 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:53.774374    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:41:53.774598 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:53.774533    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpvh" podUID="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83"
Oct 30 21:41:53.774802 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:53.774628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:41:53.774802 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:53.774713    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7d5lt" podUID="946b7964-c272-43fc-84f3-2e37be76568f"
Oct 30 21:41:53.777904 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:53.776960    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:53.777904 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:53.777075    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7hcg" podUID="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369"
Oct 30 21:41:53.842241 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:53.841185    2576 generic.go:356] "Generic (PLEG): container finished" podID="c0e7e76dbf05397217f3bc5792d5aaba" containerID="ee810f4b1c126e5ce65ff914844ac4e8c14584120f12ab8ec15de1ed4f237a27" exitCode=0
Oct 30 21:41:53.842241 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:53.841301    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal" event={"ID":"c0e7e76dbf05397217f3bc5792d5aaba","Type":"ContainerDied","Data":"ee810f4b1c126e5ce65ff914844ac4e8c14584120f12ab8ec15de1ed4f237a27"}
Oct 30 21:41:53.856724 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:53.856674    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal" event={"ID":"24d6104f1ec87a54e6dc2f912238418d","Type":"ContainerStarted","Data":"a4eaf106c0d079f3543f72fd272d8e88254e2280fdd438178a9d420deb84cbe7"}
Oct 30 21:41:54.883695 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:54.883652    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal" event={"ID":"c0e7e76dbf05397217f3bc5792d5aaba","Type":"ContainerStarted","Data":"2973a561eb94ce56df95dc897cacf1772d4e8e1621581515548129dc5cbae6d3"}
Oct 30 21:41:54.891266 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:54.891001    2576 generic.go:356] "Generic (PLEG): container finished" podID="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" containerID="13f8e566eb7194db4aa6d831def238a4360fc765a7d84b520a2bcac1f41532c4" exitCode=0
Oct 30 21:41:54.891266 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:54.891189    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l25db" event={"ID":"6eb9603c-7aa1-4d59-afc7-5fc921bf2505","Type":"ContainerDied","Data":"13f8e566eb7194db4aa6d831def238a4360fc765a7d84b520a2bcac1f41532c4"}
Oct 30 21:41:55.013301 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:55.013230    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal" podStartSLOduration=5.013165433 podStartE2EDuration="5.013165433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:54.09748329 +0000 UTC m=+6.812333598" watchObservedRunningTime="2026-10-30 21:41:55.013165433 +0000 UTC m=+7.728015739"
Oct 30 21:41:55.148850 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:55.148216    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal" podStartSLOduration=5.14817725 podStartE2EDuration="5.14817725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:55.014620988 +0000 UTC m=+7.729471288" watchObservedRunningTime="2026-10-30 21:41:55.14817725 +0000 UTC m=+7.863027567"
Oct 30 21:41:55.773660 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:55.773330    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:41:55.773660 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:55.773479    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpvh" podUID="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83"
Oct 30 21:41:55.773660 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:55.773556    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:41:55.773660 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:55.773614    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7d5lt" podUID="946b7964-c272-43fc-84f3-2e37be76568f"
Oct 30 21:41:55.774234 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:55.774053    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:55.774234 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:55.774162    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7hcg" podUID="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369"
Oct 30 21:41:55.902951 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:55.902912    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-prfrp" event={"ID":"88edf321-028a-4bb5-b729-1359bd54fb57","Type":"ContainerStarted","Data":"be5e058c3fa638cbc5d462d1e1586650266e2da3d649fb7926bb962ca7c141dd"}
Oct 30 21:41:56.908810 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:56.908634    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rr82t" event={"ID":"04aee07d-cc70-47b8-b101-d2b3b7111198","Type":"ContainerStarted","Data":"837ffcbb1129bb203565c678e77f6b7e37fd537a93921fde8a23e558fdcde1d1"}
Oct 30 21:41:57.033191 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:57.030858    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-prfrp" podStartSLOduration=3.166808657 podStartE2EDuration="8.030834916s" totalImagesPullingTime="4.864026259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:56.024846535 +0000 UTC m=+8.739696841" watchObservedRunningTime="2026-10-30 21:41:57.030834916 +0000 UTC m=+9.745685224"
Oct 30 21:41:57.486433 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:57.486332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:57.486609 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:57.486533    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:57.486609 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:57.486597    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret podName:4d1fccd5-6eaa-4a43-9a1e-d67c1005e369 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:05.486579364 +0000 UTC m=+18.201429647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret") pod "global-pull-secret-syncer-x7hcg" (UID: "4d1fccd5-6eaa-4a43-9a1e-d67c1005e369") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:41:57.587289 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:57.587242    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdm95\" (UniqueName: \"kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95\") pod \"network-check-target-7d5lt\" (UID: \"946b7964-c272-43fc-84f3-2e37be76568f\") " pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:41:57.587497 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:57.587305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:41:57.587497 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:57.587461    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:57.587614 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:57.587527    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs podName:d2f6dd07-b2ef-43b9-af3a-9bf016a45c83 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:05.587507958 +0000 UTC m=+18.302358242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs") pod "network-metrics-daemon-8kpvh" (UID: "d2f6dd07-b2ef-43b9-af3a-9bf016a45c83") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:41:57.587614 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:57.587604    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:41:57.587735 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:57.587621    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:41:57.587735 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:57.587636    2576 projected.go:196] Error preparing data for projected volume kube-api-access-sdm95 for pod openshift-network-diagnostics/network-check-target-7d5lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:57.587735 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:57.587681    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95 podName:946b7964-c272-43fc-84f3-2e37be76568f nodeName:}" failed. No retries permitted until 2026-10-30 21:42:05.587667613 +0000 UTC m=+18.302517898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdm95" (UniqueName: "kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95") pod "network-check-target-7d5lt" (UID: "946b7964-c272-43fc-84f3-2e37be76568f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:41:57.772637 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:57.772551    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:41:57.772793 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:57.772697    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpvh" podUID="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83"
Oct 30 21:41:57.774405 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:57.774357    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:41:57.774542 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:57.774483    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7d5lt" podUID="946b7964-c272-43fc-84f3-2e37be76568f"
Oct 30 21:41:57.774725 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:57.774703    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:57.774820 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:57.774792    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7hcg" podUID="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369"
Oct 30 21:41:57.914928 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:57.914876    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw" event={"ID":"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3","Type":"ContainerStarted","Data":"13410c3a369b92a642f95f3ac0d721049096fd25a1d12c95802e0c7d55f9dfbc"}
Oct 30 21:41:58.511183 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:58.511126    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:41:58.636985 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:58.636956    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:41:58.637556 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:58.637528    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:41:58.764994 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:58.763985    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rr82t" podStartSLOduration=3.857348234 podStartE2EDuration="9.7639599s" totalImagesPullingTime="5.906611666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:57.032073556 +0000 UTC m=+9.746923863" watchObservedRunningTime="2026-10-30 21:41:58.7639599 +0000 UTC m=+11.478810207"
Oct 30 21:41:58.922950 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:58.922910    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9g8ct" event={"ID":"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2","Type":"ContainerStarted","Data":"411becf563f07753b354bbfef7d8293cb3b7baf469977d0868666838f644db31"}
Oct 30 21:41:58.923494 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:58.923472    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:41:59.183416 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:59.183317    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9g8ct" podStartSLOduration=1.5828152580000001 podStartE2EDuration="10.183294797s" totalImagesPullingTime="8.600479539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:59.052026817 +0000 UTC m=+11.766877124" watchObservedRunningTime="2026-10-30 21:41:59.183294797 +0000 UTC m=+11.898145115"
Oct 30 21:41:59.385760 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:59.385734    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:41:59.766083 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:59.765984    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:41:59.385757825Z","UUID":"9edf95ff-4f74-4a28-8ebc-3ee185908309","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:41:59.768168 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:59.768140    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:41:59.768313 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:59.768177    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:41:59.772452 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:59.772427    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:41:59.772597 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:59.772427    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:41:59.772597 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:59.772572    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:41:59.772597 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:59.772574    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7d5lt" podUID="946b7964-c272-43fc-84f3-2e37be76568f"
Oct 30 21:41:59.772758 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:59.772669    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7hcg" podUID="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369"
Oct 30 21:41:59.772758 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:41:59.772749    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpvh" podUID="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83"
Oct 30 21:41:59.926758 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:59.926509    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw" event={"ID":"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3","Type":"ContainerStarted","Data":"68a78599cd8d1a8bac4c94f71f7400f9f95a982301d7eb61daa3cab309f73c6f"}
Oct 30 21:41:59.929165 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:59.929114    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9hg74" event={"ID":"f9430e64-a396-496f-a23a-96439631e58a","Type":"ContainerStarted","Data":"f5d0a9f13c94531103297522e7dd4f2a7eae6cacc55961e8a247730c1c7ba491"}
Oct 30 21:41:59.931747 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:41:59.931553    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l25db" event={"ID":"6eb9603c-7aa1-4d59-afc7-5fc921bf2505","Type":"ContainerStarted","Data":"e8f4c2720fd286999e50150d4d46da9b4906901f45a43c9092ee7ac79e42e9a9"}
Oct 30 21:42:00.163288 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:00.163218    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9hg74" podStartSLOduration=2.138476807 podStartE2EDuration="11.163196234s" totalImagesPullingTime="9.024719427s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:00.046245864 +0000 UTC m=+12.761096167" watchObservedRunningTime="2026-10-30 21:42:00.163196234 +0000 UTC m=+12.878046528"
Oct 30 21:42:00.936729 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:00.936688    2576 generic.go:356] "Generic (PLEG): container finished" podID="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" containerID="e8f4c2720fd286999e50150d4d46da9b4906901f45a43c9092ee7ac79e42e9a9" exitCode=0
Oct 30 21:42:00.937855 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:00.937786    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l25db" event={"ID":"6eb9603c-7aa1-4d59-afc7-5fc921bf2505","Type":"ContainerDied","Data":"e8f4c2720fd286999e50150d4d46da9b4906901f45a43c9092ee7ac79e42e9a9"}
Oct 30 21:42:01.773087 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:01.773064    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:01.773203 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:01.773128    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:01.773262 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:01.773249    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:01.773310 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:01.773263    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7hcg" podUID="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369"
Oct 30 21:42:01.773374 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:01.773352    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7d5lt" podUID="946b7964-c272-43fc-84f3-2e37be76568f"
Oct 30 21:42:01.773470 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:01.773454    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpvh" podUID="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83"
Oct 30 21:42:01.941176 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:01.941112    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw" event={"ID":"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3","Type":"ContainerStarted","Data":"fb27efdd25501445ddbd8b2a06836427060daafbbb324f2e589b71cfb7856bed"}
Oct 30 21:42:01.942665 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:01.942639    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4wg8g" event={"ID":"4f970ab0-3a72-4464-b767-d6867503e81d","Type":"ContainerStarted","Data":"f4904434c946c5b4887b52deee60c44e14b9cdcf27734d2922175c6c10ed77ec"}
Oct 30 21:42:01.944186 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:01.944160    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ch6rj" event={"ID":"f07039a3-5a8c-4bdf-8241-47942c8d359d","Type":"ContainerStarted","Data":"1f0930e21db830735023b3e78be8bd42a3f685eb9d87fca5c6251917857ebce8"}
Oct 30 21:42:02.066115 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:02.066007    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw" podStartSLOduration=2.159948467 podStartE2EDuration="13.06599385s" totalImagesPullingTime="10.906045383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:02.065513541 +0000 UTC m=+14.780363843" watchObservedRunningTime="2026-10-30 21:42:02.06599385 +0000 UTC m=+14.780844152"
Oct 30 21:42:02.179712 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:02.179649    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4wg8g" podStartSLOduration=3.944767353 podStartE2EDuration="13.179633291s" totalImagesPullingTime="9.234865938s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:02.179306443 +0000 UTC m=+14.894156745" watchObservedRunningTime="2026-10-30 21:42:02.179633291 +0000 UTC m=+14.894483594"
Oct 30 21:42:02.293264 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:02.293206    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ch6rj" podStartSLOduration=1.6684596379999999 podStartE2EDuration="13.293189909s" totalImagesPullingTime="11.624730271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:02.292464511 +0000 UTC m=+15.007314815" watchObservedRunningTime="2026-10-30 21:42:02.293189909 +0000 UTC m=+15.008040223"
Oct 30 21:42:02.949226 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:02.949188    2576 generic.go:356] "Generic (PLEG): container finished" podID="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" containerID="e52f208291178a3a17a945fa6429abf0777381c818237959048327e9803987bd" exitCode=0
Oct 30 21:42:02.949779 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:02.949300    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l25db" event={"ID":"6eb9603c-7aa1-4d59-afc7-5fc921bf2505","Type":"ContainerDied","Data":"e52f208291178a3a17a945fa6429abf0777381c818237959048327e9803987bd"}
Oct 30 21:42:03.773128 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:03.773099    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:03.773128 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:03.773115    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:03.773364 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:03.773110    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:03.773364 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:03.773252    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpvh" podUID="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83"
Oct 30 21:42:03.773364 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:03.773325    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7hcg" podUID="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369"
Oct 30 21:42:03.773520 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:03.773421    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7d5lt" podUID="946b7964-c272-43fc-84f3-2e37be76568f"
Oct 30 21:42:04.957318 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:04.957081    2576 generic.go:356] "Generic (PLEG): container finished" podID="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" containerID="20d42ee4018891e916d3e823da839a9d83e98648c7e148c2bce904edd5933969" exitCode=0
Oct 30 21:42:04.957318 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:04.957168    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l25db" event={"ID":"6eb9603c-7aa1-4d59-afc7-5fc921bf2505","Type":"ContainerDied","Data":"20d42ee4018891e916d3e823da839a9d83e98648c7e148c2bce904edd5933969"}
Oct 30 21:42:05.557365 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:05.557336    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:05.557527 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:05.557474    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:05.557567 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:05.557532    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret podName:4d1fccd5-6eaa-4a43-9a1e-d67c1005e369 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:21.557518826 +0000 UTC m=+34.272369120 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret") pod "global-pull-secret-syncer-x7hcg" (UID: "4d1fccd5-6eaa-4a43-9a1e-d67c1005e369") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:42:05.657903 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:05.657877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdm95\" (UniqueName: \"kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95\") pod \"network-check-target-7d5lt\" (UID: \"946b7964-c272-43fc-84f3-2e37be76568f\") " pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:05.658028 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:05.657909    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:05.658028 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:05.657999    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:05.658028 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:05.658020    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:42:05.658220 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:05.658040    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:42:05.658220 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:05.658049    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs podName:d2f6dd07-b2ef-43b9-af3a-9bf016a45c83 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:21.658035878 +0000 UTC m=+34.372886158 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs") pod "network-metrics-daemon-8kpvh" (UID: "d2f6dd07-b2ef-43b9-af3a-9bf016a45c83") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:42:05.658220 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:05.658051    2576 projected.go:196] Error preparing data for projected volume kube-api-access-sdm95 for pod openshift-network-diagnostics/network-check-target-7d5lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:05.658220 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:05.658115    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95 podName:946b7964-c272-43fc-84f3-2e37be76568f nodeName:}" failed. No retries permitted until 2026-10-30 21:42:21.658103614 +0000 UTC m=+34.372953894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sdm95" (UniqueName: "kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95") pod "network-check-target-7d5lt" (UID: "946b7964-c272-43fc-84f3-2e37be76568f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:42:05.772808 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:05.772785    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:05.772882 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:05.772785    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:05.772980 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:05.772953    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7hcg" podUID="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369"
Oct 30 21:42:05.773036 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:05.772986    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:05.773095 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:05.773059    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpvh" podUID="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83"
Oct 30 21:42:05.773166 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:05.773111    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7d5lt" podUID="946b7964-c272-43fc-84f3-2e37be76568f"
Oct 30 21:42:05.963603 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:05.963393    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 21:42:05.964027 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:05.963955    2576 generic.go:356] "Generic (PLEG): container finished" podID="0d0ce829-dc72-4f8e-91ea-2a5313fa7b11" containerID="cc45f1ac17a281a416834f923e19f5286a63595ef14bd4ac0fe424aecb98a133" exitCode=1
Oct 30 21:42:05.964027 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:05.963997    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx" event={"ID":"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11","Type":"ContainerStarted","Data":"76743bd02861ae3291cd5e1d903a0bf5a6ab3a6a5456b415aafdcaea1a3354e6"}
Oct 30 21:42:05.964117 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:05.964062    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx" event={"ID":"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11","Type":"ContainerStarted","Data":"05c7abf7709c9da12bff50899f390cde7b8a4c539f578332999b917e43979996"}
Oct 30 21:42:05.964117 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:05.964096    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx" event={"ID":"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11","Type":"ContainerStarted","Data":"490c3e53160182b83038896c64b6e8e894a6cef3988f449e4ecba877dbcf7543"}
Oct 30 21:42:05.964206 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:05.964125    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx" event={"ID":"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11","Type":"ContainerDied","Data":"cc45f1ac17a281a416834f923e19f5286a63595ef14bd4ac0fe424aecb98a133"}
Oct 30 21:42:05.964206 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:05.964172    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx" event={"ID":"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11","Type":"ContainerStarted","Data":"413c5c468ed45011037257b6185672a58afbc68823ca7a3a8137f91dfb69e2e1"}
Oct 30 21:42:06.971297 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:06.971258    2576 generic.go:356] "Generic (PLEG): container finished" podID="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" containerID="669a8406837fe61416e66bffcc660c46e4cd77a6e3618a6d335f8eaca598e6e5" exitCode=0
Oct 30 21:42:06.971803 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:06.971344    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l25db" event={"ID":"6eb9603c-7aa1-4d59-afc7-5fc921bf2505","Type":"ContainerDied","Data":"669a8406837fe61416e66bffcc660c46e4cd77a6e3618a6d335f8eaca598e6e5"}
Oct 30 21:42:06.974041 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:06.974023    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 21:42:06.974400 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:06.974380    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx" event={"ID":"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11","Type":"ContainerStarted","Data":"bef2bc165bd1ff603f9f5a4b74e3b95cd42cf1751c587384836252f680c949d6"}
Oct 30 21:42:07.773769 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:07.773738    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:07.773959 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:07.773817    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7hcg" podUID="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369"
Oct 30 21:42:07.773959 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:07.773848    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:07.774045 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:07.774018    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpvh" podUID="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83"
Oct 30 21:42:07.774091 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:07.774057    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:07.774179 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:07.774160    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7d5lt" podUID="946b7964-c272-43fc-84f3-2e37be76568f"
Oct 30 21:42:07.979910 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:07.979876    2576 generic.go:356] "Generic (PLEG): container finished" podID="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" containerID="3146e2d8e579405d73043ac31f3364939393c5a8ca973d63ac39d42a16453f38" exitCode=0
Oct 30 21:42:07.980330 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:07.979925    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l25db" event={"ID":"6eb9603c-7aa1-4d59-afc7-5fc921bf2505","Type":"ContainerDied","Data":"3146e2d8e579405d73043ac31f3364939393c5a8ca973d63ac39d42a16453f38"}
Oct 30 21:42:08.985925 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:08.985889    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l25db" event={"ID":"6eb9603c-7aa1-4d59-afc7-5fc921bf2505","Type":"ContainerStarted","Data":"0546f8fe90173c2e80fc285f05ef4e127647fa5033a4b917fd67df7ff5d4a750"}
Oct 30 21:42:08.988900 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:08.988880    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 21:42:08.989274 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:08.989250    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx" event={"ID":"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11","Type":"ContainerStarted","Data":"38cc75f3b83fcb679f54b5af9d363328bfbd1327450ffc8c7ff73ac0e2f10c26"}
Oct 30 21:42:09.110997 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:09.110948    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l25db" podStartSLOduration=7.364614973 podStartE2EDuration="20.110935426s" totalImagesPullingTime="12.746320453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:09.110772706 +0000 UTC m=+21.825623009" watchObservedRunningTime="2026-10-30 21:42:09.110935426 +0000 UTC m=+21.825785729"
Oct 30 21:42:09.772858 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:09.772830    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:09.773050 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:09.772931    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7hcg" podUID="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369"
Oct 30 21:42:09.773050 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:09.772946    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:09.773050 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:09.772843    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:09.773050 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:09.773025    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7d5lt" podUID="946b7964-c272-43fc-84f3-2e37be76568f"
Oct 30 21:42:09.773260 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:09.773155    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpvh" podUID="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83"
Oct 30 21:42:10.996659 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:10.996484    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 21:42:10.997203 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:10.996997    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx" event={"ID":"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11","Type":"ContainerStarted","Data":"a7ec384f392a313cd5671f5c318fb9c5d485a16f9ec5fb8616f6276c8820f01c"}
Oct 30 21:42:10.997361 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:10.997339    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:10.997435 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:10.997403    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:10.997470 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:10.997451    2576 scope.go:122] "RemoveContainer" containerID="cc45f1ac17a281a416834f923e19f5286a63595ef14bd4ac0fe424aecb98a133"
Oct 30 21:42:11.014356 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:11.014329    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:11.772808 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:11.772763    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:11.773054 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:11.772819    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:11.773054 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:11.772825    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:11.773054 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:11.772902    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpvh" podUID="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83"
Oct 30 21:42:11.773054 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:11.772995    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7hcg" podUID="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369"
Oct 30 21:42:11.773054 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:11.773050    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7d5lt" podUID="946b7964-c272-43fc-84f3-2e37be76568f"
Oct 30 21:42:12.003536 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:12.003508    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 21:42:12.003986 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:12.003860    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx" event={"ID":"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11","Type":"ContainerStarted","Data":"27d7bbdab0a1d67400a4e09fadd74a20e47d10773424523a83370b8dae07a4c0"}
Oct 30 21:42:12.004096 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:12.004059    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:12.020014 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:12.019989    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:12.204975 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:12.204897    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx" podStartSLOduration=7.7595444 podStartE2EDuration="23.204876752s" totalImagesPullingTime="15.445332352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:12.183666301 +0000 UTC m=+24.898516602" watchObservedRunningTime="2026-10-30 21:42:12.204876752 +0000 UTC m=+24.919727061"
Oct 30 21:42:12.831519 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:12.831486    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x7hcg"]
Oct 30 21:42:12.831695 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:12.831605    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:12.831735 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:12.831703    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7hcg" podUID="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369"
Oct 30 21:42:12.834129 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:12.834096    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7d5lt"]
Oct 30 21:42:12.834291 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:12.834220    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:12.834363 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:12.834315    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7d5lt" podUID="946b7964-c272-43fc-84f3-2e37be76568f"
Oct 30 21:42:12.835345 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:12.835046    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8kpvh"]
Oct 30 21:42:12.835345 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:12.835191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:12.835345 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:12.835306    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpvh" podUID="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83"
Oct 30 21:42:14.773123 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:14.773089    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:14.773123 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:14.773111    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:14.773587 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:14.773245    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpvh" podUID="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83"
Oct 30 21:42:14.773587 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:14.773300    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:14.773587 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:14.773420    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7hcg" podUID="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369"
Oct 30 21:42:14.773587 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:14.773535    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7d5lt" podUID="946b7964-c272-43fc-84f3-2e37be76568f"
Oct 30 21:42:16.772439 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:16.772407    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:16.772439 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:16.772420    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:16.772945 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:16.772408    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:16.772945 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:16.772553    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7hcg" podUID="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369"
Oct 30 21:42:16.772945 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:16.772603    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7d5lt" podUID="946b7964-c272-43fc-84f3-2e37be76568f"
Oct 30 21:42:16.772945 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:16.772673    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpvh" podUID="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83"
Oct 30 21:42:18.547843 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.547663    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeReady"
Oct 30 21:42:18.548219 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.547956    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:42:18.665976 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.665944    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x6dnl"]
Oct 30 21:42:18.668044 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.668020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:18.715368 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.715256    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9tvgl"]
Oct 30 21:42:18.717233 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.717212    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9tvgl"
Oct 30 21:42:18.720259 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.720234    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:42:18.720454 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.720436    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:42:18.720524 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.720508    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j7s9f\""
Oct 30 21:42:18.744502 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.744469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a392100e-84e5-4329-8e2c-6257b15fc58a-config-volume\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:18.744640 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.744517    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a392100e-84e5-4329-8e2c-6257b15fc58a-tmp-dir\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:18.744640 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.744539    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:18.744727 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.744643    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn7pc\" (UniqueName: \"kubernetes.io/projected/a392100e-84e5-4329-8e2c-6257b15fc58a-kube-api-access-cn7pc\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:18.748046 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.748020    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x6dnl"]
Oct 30 21:42:18.772249 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.772217    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:18.772391 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.772300    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:18.772441 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.772392    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:18.775648 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.775615    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9tvgl"]
Oct 30 21:42:18.820268 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.820241    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:42:18.820412 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.820394    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:42:18.820636 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.820621    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:42:18.820666 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.820623    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7xfbb\""
Oct 30 21:42:18.845426 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.845398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l49w6\" (UniqueName: \"kubernetes.io/projected/49253fd8-a9b7-457a-8104-1df48059e145-kube-api-access-l49w6\") pod \"ingress-canary-9tvgl\" (UID: \"49253fd8-a9b7-457a-8104-1df48059e145\") " pod="openshift-ingress-canary/ingress-canary-9tvgl"
Oct 30 21:42:18.845562 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.845456    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a392100e-84e5-4329-8e2c-6257b15fc58a-tmp-dir\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:18.845596 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.845562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:18.845632 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.845601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn7pc\" (UniqueName: \"kubernetes.io/projected/a392100e-84e5-4329-8e2c-6257b15fc58a-kube-api-access-cn7pc\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:18.845695 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.845680    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a392100e-84e5-4329-8e2c-6257b15fc58a-config-volume\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:18.845738 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:18.845688    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:18.845768 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:18.845756    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls podName:a392100e-84e5-4329-8e2c-6257b15fc58a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:19.345736768 +0000 UTC m=+32.060587064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls") pod "dns-default-x6dnl" (UID: "a392100e-84e5-4329-8e2c-6257b15fc58a") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:18.845840 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.845818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49253fd8-a9b7-457a-8104-1df48059e145-cert\") pod \"ingress-canary-9tvgl\" (UID: \"49253fd8-a9b7-457a-8104-1df48059e145\") " pod="openshift-ingress-canary/ingress-canary-9tvgl"
Oct 30 21:42:18.845937 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.845920    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a392100e-84e5-4329-8e2c-6257b15fc58a-tmp-dir\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:18.846212 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.846194    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a392100e-84e5-4329-8e2c-6257b15fc58a-config-volume\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:18.864856 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.864833    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nx7wb"]
Oct 30 21:42:18.866595 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.866582    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:18.904666 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.904621    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn7pc\" (UniqueName: \"kubernetes.io/projected/a392100e-84e5-4329-8e2c-6257b15fc58a-kube-api-access-cn7pc\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:18.920245 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.920203    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:42:18.920245 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.920215    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:42:18.920245 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.920241    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:42:18.920494 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.920311    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:42:18.920494 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.920320    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:42:18.920634 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.920617    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:42:18.920932 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.920914    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:42:18.921109 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.921096    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9csh8\""
Oct 30 21:42:18.921250 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.921237    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:42:18.921285 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.921264    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nrkss\""
Oct 30 21:42:18.921285 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.921266    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wcr9d\""
Oct 30 21:42:18.921456 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.921299    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sflz7\""
Oct 30 21:42:18.925187 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.925158    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nx7wb"]
Oct 30 21:42:18.946533 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.946492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:18.946712 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.946631    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/00223ac9-d79b-42d2-ae49-0f17451b2228-data-volume\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:18.946712 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.946665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:18.946840 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.946727    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/00223ac9-d79b-42d2-ae49-0f17451b2228-crio-socket\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:18.946918 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.946899    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49253fd8-a9b7-457a-8104-1df48059e145-cert\") pod \"ingress-canary-9tvgl\" (UID: \"49253fd8-a9b7-457a-8104-1df48059e145\") " pod="openshift-ingress-canary/ingress-canary-9tvgl"
Oct 30 21:42:18.946971 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.946927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn64w\" (UniqueName: \"kubernetes.io/projected/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-api-access-bn64w\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:18.946971 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.946952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l49w6\" (UniqueName: \"kubernetes.io/projected/49253fd8-a9b7-457a-8104-1df48059e145-kube-api-access-l49w6\") pod \"ingress-canary-9tvgl\" (UID: \"49253fd8-a9b7-457a-8104-1df48059e145\") " pod="openshift-ingress-canary/ingress-canary-9tvgl"
Oct 30 21:42:18.949766 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:18.949744    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49253fd8-a9b7-457a-8104-1df48059e145-cert\") pod \"ingress-canary-9tvgl\" (UID: \"49253fd8-a9b7-457a-8104-1df48059e145\") " pod="openshift-ingress-canary/ingress-canary-9tvgl"
Oct 30 21:42:19.004914 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.004849    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l49w6\" (UniqueName: \"kubernetes.io/projected/49253fd8-a9b7-457a-8104-1df48059e145-kube-api-access-l49w6\") pod \"ingress-canary-9tvgl\" (UID: \"49253fd8-a9b7-457a-8104-1df48059e145\") " pod="openshift-ingress-canary/ingress-canary-9tvgl"
Oct 30 21:42:19.027609 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.027581    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9tvgl"
Oct 30 21:42:19.047621 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.047590    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/00223ac9-d79b-42d2-ae49-0f17451b2228-crio-socket\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:19.047741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.047646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn64w\" (UniqueName: \"kubernetes.io/projected/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-api-access-bn64w\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:19.047741 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.047677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:19.047844 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.047743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/00223ac9-d79b-42d2-ae49-0f17451b2228-data-volume\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:19.047844 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.047774    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:19.047844 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.047825    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/00223ac9-d79b-42d2-ae49-0f17451b2228-crio-socket\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:19.047983 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:19.047882    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:19.047983 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:19.047912    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:19.047983 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:19.047969    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:19.547950118 +0000 UTC m=+32.262800412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:19.048090 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:19.047988    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:19.547977417 +0000 UTC m=+32.262827701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:19.048128 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.048107    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/00223ac9-d79b-42d2-ae49-0f17451b2228-data-volume\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:19.105591 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.105564    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn64w\" (UniqueName: \"kubernetes.io/projected/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-api-access-bn64w\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:19.114444 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.114416    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-fpxlc-s8s8f"]
Oct 30 21:42:19.116699 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.116683    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:19.169953 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.169927    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xlldj\""
Oct 30 21:42:19.170388 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.170373    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:42:19.219706 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.219679    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:42:19.228059 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.228031    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fpxlc-s8s8f"]
Oct 30 21:42:19.249398 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.249367    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d6fv\" (UniqueName: \"kubernetes.io/projected/47734525-f46d-41de-93b8-c8b8f4087651-kube-api-access-2d6fv\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:19.249398 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.249402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-trusted-ca-bundle\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:19.249586 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.249423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/47734525-f46d-41de-93b8-c8b8f4087651-archives-path\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:19.249617 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.249580    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-service-ca-bundle\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:19.350343 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.350313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d6fv\" (UniqueName: \"kubernetes.io/projected/47734525-f46d-41de-93b8-c8b8f4087651-kube-api-access-2d6fv\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:19.350509 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.350352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-trusted-ca-bundle\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:19.350509 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.350373    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/47734525-f46d-41de-93b8-c8b8f4087651-archives-path\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:19.350616 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.350590    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:19.350664 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.350633    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-service-ca-bundle\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:19.350831 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:19.350811    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:19.350917 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:19.350905    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls podName:a392100e-84e5-4329-8e2c-6257b15fc58a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.350879498 +0000 UTC m=+33.065729777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls") pod "dns-default-x6dnl" (UID: "a392100e-84e5-4329-8e2c-6257b15fc58a") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:19.350962 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.350933    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/47734525-f46d-41de-93b8-c8b8f4087651-archives-path\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:19.351374 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.351358    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-service-ca-bundle\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:19.351534 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.351516    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-trusted-ca-bundle\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:19.405580 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.405554    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9tvgl"]
Oct 30 21:42:19.409383 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.409361    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d6fv\" (UniqueName: \"kubernetes.io/projected/47734525-f46d-41de-93b8-c8b8f4087651-kube-api-access-2d6fv\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:19.410010 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:42:19.409991    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49253fd8_a9b7_457a_8104_1df48059e145.slice/crio-49c5bcdd64661713256a5b41de3be06730c40a54c824eb52061aa389a4ec6dd4 WatchSource:0}: Error finding container 49c5bcdd64661713256a5b41de3be06730c40a54c824eb52061aa389a4ec6dd4: Status 404 returned error can't find the container with id 49c5bcdd64661713256a5b41de3be06730c40a54c824eb52061aa389a4ec6dd4
Oct 30 21:42:19.427423 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.427399    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:19.552756 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.552724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:19.553651 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.552789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:19.553651 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:19.552866    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:19.553651 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:19.552896    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:19.553651 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:19.552942    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.552918361 +0000 UTC m=+33.267768649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:19.553651 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:19.552958    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:20.552950741 +0000 UTC m=+33.267801023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:19.797930 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:19.797852    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fpxlc-s8s8f"]
Oct 30 21:42:19.801343 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:42:19.801311    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47734525_f46d_41de_93b8_c8b8f4087651.slice/crio-5fc53b1e96939496d00537dbc80bfc2a02d885734e207857daf817cb2d7643b3 WatchSource:0}: Error finding container 5fc53b1e96939496d00537dbc80bfc2a02d885734e207857daf817cb2d7643b3: Status 404 returned error can't find the container with id 5fc53b1e96939496d00537dbc80bfc2a02d885734e207857daf817cb2d7643b3
Oct 30 21:42:20.020781 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.020744    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9tvgl" event={"ID":"49253fd8-a9b7-457a-8104-1df48059e145","Type":"ContainerStarted","Data":"49c5bcdd64661713256a5b41de3be06730c40a54c824eb52061aa389a4ec6dd4"}
Oct 30 21:42:20.021881 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.021862    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fpxlc-s8s8f" event={"ID":"47734525-f46d-41de-93b8-c8b8f4087651","Type":"ContainerStarted","Data":"5fc53b1e96939496d00537dbc80bfc2a02d885734e207857daf817cb2d7643b3"}
Oct 30 21:42:20.359635 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.359599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:20.359794 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:20.359764    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:20.359846 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:20.359836    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls podName:a392100e-84e5-4329-8e2c-6257b15fc58a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:22.359817891 +0000 UTC m=+35.074668175 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls") pod "dns-default-x6dnl" (UID: "a392100e-84e5-4329-8e2c-6257b15fc58a") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:20.419884 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.419847    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-p2t8s"]
Oct 30 21:42:20.422513 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.422473    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p2t8s"
Oct 30 21:42:20.474381 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.474350    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dzgdj\""
Oct 30 21:42:20.474561 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.474448    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:20.474561 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.474515    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:42:20.478683 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.478658    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-p2t8s"]
Oct 30 21:42:20.560696 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.560654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:20.561224 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.560762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25zcv\" (UniqueName: \"kubernetes.io/projected/c370a2e7-4648-47ed-ae04-4d6d03ee323e-kube-api-access-25zcv\") pod \"migrator-66df5d496b-p2t8s\" (UID: \"c370a2e7-4648-47ed-ae04-4d6d03ee323e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p2t8s"
Oct 30 21:42:20.561224 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:20.560775    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:20.561224 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.560799    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:20.561224 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:20.560847    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:22.560828504 +0000 UTC m=+35.275678807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:20.561224 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:20.560962    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:20.561224 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:20.561039    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:22.561019667 +0000 UTC m=+35.275869963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:20.661614 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.661532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25zcv\" (UniqueName: \"kubernetes.io/projected/c370a2e7-4648-47ed-ae04-4d6d03ee323e-kube-api-access-25zcv\") pod \"migrator-66df5d496b-p2t8s\" (UID: \"c370a2e7-4648-47ed-ae04-4d6d03ee323e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p2t8s"
Oct 30 21:42:20.694095 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.694061    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-pbghc"]
Oct 30 21:42:20.698052 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.698024    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:20.718276 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.718239    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25zcv\" (UniqueName: \"kubernetes.io/projected/c370a2e7-4648-47ed-ae04-4d6d03ee323e-kube-api-access-25zcv\") pod \"migrator-66df5d496b-p2t8s\" (UID: \"c370a2e7-4648-47ed-ae04-4d6d03ee323e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p2t8s"
Oct 30 21:42:20.735097 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.735059    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p2t8s"
Oct 30 21:42:20.755794 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.755327    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 21:42:20.755794 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.755470    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 21:42:20.755794 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.755545    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 21:42:20.755794 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.755645    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5fg22\""
Oct 30 21:42:20.755794 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.755740    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 21:42:20.756484 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.756459    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-pbghc"]
Oct 30 21:42:20.863291 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.863256    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/937068c6-0ce1-4485-afc1-0c42a8a16aba-signing-cabundle\") pod \"service-ca-5fffc97bdf-pbghc\" (UID: \"937068c6-0ce1-4485-afc1-0c42a8a16aba\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:20.863470 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.863333    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrxc\" (UniqueName: \"kubernetes.io/projected/937068c6-0ce1-4485-afc1-0c42a8a16aba-kube-api-access-dlrxc\") pod \"service-ca-5fffc97bdf-pbghc\" (UID: \"937068c6-0ce1-4485-afc1-0c42a8a16aba\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:20.863470 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.863448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/937068c6-0ce1-4485-afc1-0c42a8a16aba-signing-key\") pod \"service-ca-5fffc97bdf-pbghc\" (UID: \"937068c6-0ce1-4485-afc1-0c42a8a16aba\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:20.965066 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.964974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrxc\" (UniqueName: \"kubernetes.io/projected/937068c6-0ce1-4485-afc1-0c42a8a16aba-kube-api-access-dlrxc\") pod \"service-ca-5fffc97bdf-pbghc\" (UID: \"937068c6-0ce1-4485-afc1-0c42a8a16aba\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:20.965066 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.965026    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/937068c6-0ce1-4485-afc1-0c42a8a16aba-signing-key\") pod \"service-ca-5fffc97bdf-pbghc\" (UID: \"937068c6-0ce1-4485-afc1-0c42a8a16aba\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:20.965276 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.965237    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/937068c6-0ce1-4485-afc1-0c42a8a16aba-signing-cabundle\") pod \"service-ca-5fffc97bdf-pbghc\" (UID: \"937068c6-0ce1-4485-afc1-0c42a8a16aba\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:20.966010 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.965988    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/937068c6-0ce1-4485-afc1-0c42a8a16aba-signing-cabundle\") pod \"service-ca-5fffc97bdf-pbghc\" (UID: \"937068c6-0ce1-4485-afc1-0c42a8a16aba\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:20.968436 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:20.968417    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/937068c6-0ce1-4485-afc1-0c42a8a16aba-signing-key\") pod \"service-ca-5fffc97bdf-pbghc\" (UID: \"937068c6-0ce1-4485-afc1-0c42a8a16aba\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:21.026791 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:21.026758    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrxc\" (UniqueName: \"kubernetes.io/projected/937068c6-0ce1-4485-afc1-0c42a8a16aba-kube-api-access-dlrxc\") pod \"service-ca-5fffc97bdf-pbghc\" (UID: \"937068c6-0ce1-4485-afc1-0c42a8a16aba\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:21.118348 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:21.118295    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-p2t8s"]
Oct 30 21:42:21.124285 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:42:21.124256    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc370a2e7_4648_47ed_ae04_4d6d03ee323e.slice/crio-c978e4740588cc682044f40fc8939dd3907c788d3e687cd192d8915277d23f0f WatchSource:0}: Error finding container c978e4740588cc682044f40fc8939dd3907c788d3e687cd192d8915277d23f0f: Status 404 returned error can't find the container with id c978e4740588cc682044f40fc8939dd3907c788d3e687cd192d8915277d23f0f
Oct 30 21:42:21.313636 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:21.313323    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:21.572328 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:21.572249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:21.575836 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:21.575769    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:21.673362 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:21.673321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdm95\" (UniqueName: \"kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95\") pod \"network-check-target-7d5lt\" (UID: \"946b7964-c272-43fc-84f3-2e37be76568f\") " pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:21.673362 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:21.673365    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:21.673576 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:21.673482    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:42:21.673576 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:21.673547    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs podName:d2f6dd07-b2ef-43b9-af3a-9bf016a45c83 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:53.673531138 +0000 UTC m=+66.388381437 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs") pod "network-metrics-daemon-8kpvh" (UID: "d2f6dd07-b2ef-43b9-af3a-9bf016a45c83") : secret "metrics-daemon-secret" not found
Oct 30 21:42:21.676295 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:21.676270    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdm95\" (UniqueName: \"kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95\") pod \"network-check-target-7d5lt\" (UID: \"946b7964-c272-43fc-84f3-2e37be76568f\") " pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:21.695731 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:21.695696    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-pbghc"]
Oct 30 21:42:21.700657 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:42:21.700616    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937068c6_0ce1_4485_afc1_0c42a8a16aba.slice/crio-15e6800bd921632283fb5a0ab9ad47002b7144c9d49e0137d1826f30c14dd4af WatchSource:0}: Error finding container 15e6800bd921632283fb5a0ab9ad47002b7144c9d49e0137d1826f30c14dd4af: Status 404 returned error can't find the container with id 15e6800bd921632283fb5a0ab9ad47002b7144c9d49e0137d1826f30c14dd4af
Oct 30 21:42:21.784431 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:21.784402    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:21.790724 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:21.790690    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:22.031302 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:22.031267    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc" event={"ID":"937068c6-0ce1-4485-afc1-0c42a8a16aba","Type":"ContainerStarted","Data":"15e6800bd921632283fb5a0ab9ad47002b7144c9d49e0137d1826f30c14dd4af"}
Oct 30 21:42:22.032788 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:22.032746    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p2t8s" event={"ID":"c370a2e7-4648-47ed-ae04-4d6d03ee323e","Type":"ContainerStarted","Data":"c978e4740588cc682044f40fc8939dd3907c788d3e687cd192d8915277d23f0f"}
Oct 30 21:42:22.166890 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:22.166803    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x7hcg"]
Oct 30 21:42:22.175705 ip-10-0-133-226 kubenswrapper[2576]: W1030 21:42:22.170833    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1fccd5_6eaa_4a43_9a1e_d67c1005e369.slice/crio-aa91d14fe097a8fffacc20e7b97ea5f6c7887ddefd3b1c21348c548657fc392c WatchSource:0}: Error finding container aa91d14fe097a8fffacc20e7b97ea5f6c7887ddefd3b1c21348c548657fc392c: Status 404 returned error can't find the container with id aa91d14fe097a8fffacc20e7b97ea5f6c7887ddefd3b1c21348c548657fc392c
Oct 30 21:42:22.221375 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:22.221291    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7d5lt"]
Oct 30 21:42:22.379521 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:22.379483    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:22.379703 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:22.379682    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:22.379798 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:22.379756    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls podName:a392100e-84e5-4329-8e2c-6257b15fc58a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:26.379736552 +0000 UTC m=+39.094586835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls") pod "dns-default-x6dnl" (UID: "a392100e-84e5-4329-8e2c-6257b15fc58a") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:22.580975 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:22.580946    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:22.581393 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:22.581013    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:22.581393 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:22.581086    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:22.581393 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:22.581150    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:22.581393 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:22.581180    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:26.581157628 +0000 UTC m=+39.296007911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:22.581393 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:22.581204    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:26.581188034 +0000 UTC m=+39.296038315 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:23.036680 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:23.036642    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7d5lt" event={"ID":"946b7964-c272-43fc-84f3-2e37be76568f","Type":"ContainerStarted","Data":"4d86251e01020a78a38815518237105966945da8f412b835ec16334fb49fa2ed"}
Oct 30 21:42:23.038309 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:23.038278    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fpxlc-s8s8f" event={"ID":"47734525-f46d-41de-93b8-c8b8f4087651","Type":"ContainerStarted","Data":"371a295b33c21af65143447f555f780b7ddee037bfda841824c6efbddb79883f"}
Oct 30 21:42:23.039935 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:23.039911    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x7hcg" event={"ID":"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369","Type":"ContainerStarted","Data":"aa91d14fe097a8fffacc20e7b97ea5f6c7887ddefd3b1c21348c548657fc392c"}
Oct 30 21:42:23.151515 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:23.151459    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-fpxlc-s8s8f" podStartSLOduration=1.3813746089999999 podStartE2EDuration="4.151438507s" totalImagesPullingTime="2.770063898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:23.150753318 +0000 UTC m=+35.865603639" watchObservedRunningTime="2026-10-30 21:42:23.151438507 +0000 UTC m=+35.866288812"
Oct 30 21:42:24.045579 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:24.045072    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p2t8s" event={"ID":"c370a2e7-4648-47ed-ae04-4d6d03ee323e","Type":"ContainerStarted","Data":"45f000130ea360d8afa3d22b6fe42761ecc25f613e4c410d1a3d0a8b6cbe2b5f"}
Oct 30 21:42:24.047893 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:24.047857    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9tvgl" event={"ID":"49253fd8-a9b7-457a-8104-1df48059e145","Type":"ContainerStarted","Data":"544f0aee5415172fcb355c8cd6312b7c5ce94a157e14ccc74caf77723e798b96"}
Oct 30 21:42:24.159354 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:24.159287    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p2t8s" podStartSLOduration=2.163655362 podStartE2EDuration="4.159269558s" totalImagesPullingTime="1.995614196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:24.158519561 +0000 UTC m=+36.873369864" watchObservedRunningTime="2026-10-30 21:42:24.159269558 +0000 UTC m=+36.874119923"
Oct 30 21:42:24.275602 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:24.275542    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9tvgl" podStartSLOduration=2.309972906 podStartE2EDuration="6.27551925s" totalImagesPullingTime="3.965546344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:24.274490771 +0000 UTC m=+36.989341073" watchObservedRunningTime="2026-10-30 21:42:24.27551925 +0000 UTC m=+36.990369554"
Oct 30 21:42:26.056279 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:26.056236    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc" event={"ID":"937068c6-0ce1-4485-afc1-0c42a8a16aba","Type":"ContainerStarted","Data":"e674015f6116130190f413a29c5c36c419c9a4e474f157facc874376554798c7"}
Oct 30 21:42:26.170721 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:26.170661    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc" podStartSLOduration=2.7576671189999997 podStartE2EDuration="6.170647493s" totalImagesPullingTime="3.412980374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:26.170323633 +0000 UTC m=+38.885173937" watchObservedRunningTime="2026-10-30 21:42:26.170647493 +0000 UTC m=+38.885497818"
Oct 30 21:42:26.411164 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:26.411109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:26.411321 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:26.411249    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:42:26.411321 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:26.411304    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls podName:a392100e-84e5-4329-8e2c-6257b15fc58a nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.41129133 +0000 UTC m=+47.126141623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls") pod "dns-default-x6dnl" (UID: "a392100e-84e5-4329-8e2c-6257b15fc58a") : secret "dns-default-metrics-tls" not found
Oct 30 21:42:26.612679 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:26.612645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:26.612832 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:26.612710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:26.612832 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:26.612766    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:26.613032 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:26.612834    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.612818941 +0000 UTC m=+47.327669225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:26.613032 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:26.612840    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:26.613032 ip-10-0-133-226 kubenswrapper[2576]: E1030 21:42:26.612886    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.612869348 +0000 UTC m=+47.327719649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:26.692587 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:26.692500    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9hg74_f9430e64-a396-496f-a23a-96439631e58a/dns-node-resolver/0.log"
Oct 30 21:42:27.061192 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:27.061087    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7d5lt" event={"ID":"946b7964-c272-43fc-84f3-2e37be76568f","Type":"ContainerStarted","Data":"08c6595d8cd0aa8077be8fec4c98e529318ab0f3fdc8e0717e79f829f8831f37"}
Oct 30 21:42:27.176919 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:27.176856    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7d5lt" podStartSLOduration=33.622424434 podStartE2EDuration="38.176834702s" totalImagesPullingTime="4.554410268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:27.175631824 +0000 UTC m=+39.890482145" watchObservedRunningTime="2026-10-30 21:42:27.176834702 +0000 UTC m=+39.891685008"
Oct 30 21:42:27.893886 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:27.893855    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rr82t_04aee07d-cc70-47b8-b101-d2b3b7111198/node-ca/0.log"
Oct 30 21:42:28.064186 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:28.064154    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:28.293031 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:28.292961    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9tvgl_49253fd8-a9b7-457a-8104-1df48059e145/serve-healthcheck-canary/0.log"
Oct 30 21:42:28.893336 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:28.893313    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-p2t8s_c370a2e7-4648-47ed-ae04-4d6d03ee323e/migrator/0.log"
Oct 30 21:42:29.070018 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:29.069978    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x7hcg" event={"ID":"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369","Type":"ContainerStarted","Data":"d2b8b07296f5c1127218028041f73dfb1bf405f262bdadfa0780b57186d75c8c"}
Oct 30 21:42:29.186036 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:29.185931    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x7hcg" podStartSLOduration=33.484237788 podStartE2EDuration="40.185909539s" totalImagesPullingTime="6.701671751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:29.18481894 +0000 UTC m=+41.899669243" watchObservedRunningTime="2026-10-30 21:42:29.185909539 +0000 UTC m=+41.900759841"
Oct 30 21:42:29.186420 ip-10-0-133-226 kubenswrapper[2576]: I1030 21:42:29.186340    2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:42:29.186550 ip-10-0-133-226 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:42:29.205573 ip-10-0-133-226 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:42:29.205748 ip-10-0-133-226 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:42:29.205934 ip-10-0-133-226 systemd[1]: kubelet.service: Consumed 2.691s CPU time.
Oct 30 21:42:29.221669 ip-10-0-133-226 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:42:29.311058 ip-10-0-133-226 kubenswrapper[4553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:42:29.311058 ip-10-0-133-226 kubenswrapper[4553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:42:29.311058 ip-10-0-133-226 kubenswrapper[4553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:42:29.311058 ip-10-0-133-226 kubenswrapper[4553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:42:29.313120 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313036    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:29.313120 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313122    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313126    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313146    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313151    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313156    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313161    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313165    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313169    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313173    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313177    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313180    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313183    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313186    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313188    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313192    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313194    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313197    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313199    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313202    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313205    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:29.313211 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313207    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313210    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313212    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313215    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313218    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313221    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313224    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313227    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313229    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313232    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313234    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313237    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313239    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313242    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313244    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313247    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313249    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313252    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313254    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:29.313720 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313257    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313260    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313263    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313265    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313268    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313270    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313274    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313276    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313279    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313281    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313284    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313286    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313289    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313292    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313294    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313298    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313300    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313303    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313306    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:29.314207 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313308    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313310    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313313    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313316    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313318    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313321    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313323    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313326    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313328    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313333    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313338    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313341    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313344    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313348    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313350    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313353    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313355    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313358    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313361    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:29.314684 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313364    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313366    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313369    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313372    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313374    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313377    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313380    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313382    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313385    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313388    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313390    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313393    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313396    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313398    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313401    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313404    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313406    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313409    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313411    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313415    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:29.315178 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313419    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313422    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313424    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313427    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313429    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313431    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313434    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.313447    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314256    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314263    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314266    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314269    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314273    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314276    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314278    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314281    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314284    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314286    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314289    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314291    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:29.315680 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314294    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314296    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314299    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314302    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314305    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314307    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314310    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314312    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314315    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314317    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314319    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314322    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314324    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314327    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314329    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314332    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314334    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314337    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314340    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314345    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:29.316183 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314348    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314351    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314353    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314356    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314359    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314361    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314364    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314366    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314369    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314371    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314375    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314379    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314381    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314385    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314387    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314390    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314392    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314395    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314397    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:29.316677 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314400    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314402    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314405    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314407    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314410    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314413    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314415    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314418    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314420    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314423    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314425    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314428    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314430    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314434    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314437    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314440    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314443    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314445    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314448    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:29.317218 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314451    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314453    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314456    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314459    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314463    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314466    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314469    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314473    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314475    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314478    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314480    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314483    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314485    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314488    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314490    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314493    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314495    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314498    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314500    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:29.317702 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314502    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314505    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314507    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314510    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314512    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314515    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314518    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314520    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314523    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314526    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314529    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314531    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314534    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314536    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314539    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314541    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.314545    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315031    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315036    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315039    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:29.318181 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315041    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315044    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315046    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315049    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315052    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315055    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315057    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315060    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315063    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315065    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315068    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315070    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315072    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315075    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315089    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315092    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315095    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315097    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315100    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315102    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:29.318670 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315105    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315109    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315112    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315115    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315118    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315120    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315123    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315125    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315128    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315144    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315147    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315150    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315152    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315155    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315157    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315160    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315163    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315165    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315168    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:29.319223 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315171    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315173    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315176    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315179    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315183    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315186    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315195    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315198    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315201    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315204    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315206    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315209    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315212    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315214    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315217    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315220    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315223    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315226    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315228    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:29.319910 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315231    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315234    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315236    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315239    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315241    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315245    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315248    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315251    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315254    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315257    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315259    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315262    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315264    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315267    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315269    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315272    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315275    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315277    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315279    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:29.320779 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315283    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315286    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315288    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315291    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315293    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315296    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315298    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315301    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315304    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315306    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315309    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315312    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315314    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315316    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315319    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315322    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315324    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315327    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315329    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315332    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:29.321379 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315334    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315337    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315340    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315342    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315345    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.315347    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315446    4553 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315456    4553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315466    4553 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315473    4553 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315479    4553 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315483    4553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315490    4553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315495    4553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315498    4553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315502    4553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315506    4553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315509    4553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315512    4553 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315516    4553 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315518    4553 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315522    4553 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315524    4553 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:42:29.322033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315528    4553 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315532    4553 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315535    4553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315538    4553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315541    4553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315545    4553 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315549    4553 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315551    4553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315555    4553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315558    4553 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315561    4553 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315564    4553 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315568    4553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315572    4553 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315574    4553 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315580    4553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315583    4553 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315586    4553 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315589    4553 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315592    4553 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315595    4553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315602    4553 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315605    4553 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315608    4553 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:42:29.322647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315611    4553 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315614    4553 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315618    4553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315621    4553 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315624    4553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315627    4553 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315630    4553 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315633    4553 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315636    4553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315640    4553 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315643    4553 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315646    4553 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315649    4553 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315653    4553 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315657    4553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315660    4553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315663    4553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315666    4553 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315669    4553 flags.go:64] FLAG: --help="false"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315672    4553 flags.go:64] FLAG: --hostname-override="ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315676    4553 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315679    4553 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315683    4553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:42:29.323316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315687    4553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315691    4553 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315694    4553 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315697    4553 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315700    4553 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315703    4553 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315707    4553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315710    4553 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315714    4553 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315717    4553 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315720    4553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315723    4553 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315726    4553 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315728    4553 flags.go:64] FLAG: --lock-file=""
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315731    4553 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315735    4553 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315738    4553 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315743    4553 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315746    4553 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315750    4553 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315753    4553 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315756    4553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315760    4553 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315762    4553 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315765    4553 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:42:29.323863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315770    4553 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315773    4553 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315777    4553 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315780    4553 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315783    4553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315786    4553 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315790    4553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315793    4553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315796    4553 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315799    4553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315806    4553 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315809    4553 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315813    4553 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315817    4553 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315820    4553 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315823    4553 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315826    4553 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315829    4553 flags.go:64] FLAG: --port="10250"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315832    4553 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315835    4553 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0856a38c16b2df750"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315838    4553 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315841    4553 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315844    4553 flags.go:64] FLAG: --register-node="true"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315847    4553 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315851    4553 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315853    4553 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:42:29.324469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315856    4553 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315860    4553 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315864    4553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315867    4553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315870    4553 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315873    4553 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315876    4553 flags.go:64] FLAG: --runonce="false"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315878    4553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315881    4553 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315884    4553 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315887    4553 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315890    4553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315894    4553 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315897    4553 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315900    4553 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315903    4553 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315906    4553 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315909    4553 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315912    4553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315917    4553 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315920    4553 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315923    4553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315928    4553 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315931    4553 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315934    4553 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:42:29.325100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315938    4553 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315941    4553 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315944    4553 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315947    4553 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315950    4553 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315953    4553 flags.go:64] FLAG: --v="2"
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315957    4553 flags.go:64] FLAG: --version="false"
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315961    4553 flags.go:64] FLAG: --vmodule=""
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315966    4553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.315969    4553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316105    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316108    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316111    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316114    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316117    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316119    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316122    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316125    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316127    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316145    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316148    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316151    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:29.325744 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316153    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316156    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316159    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316162    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316166    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316168    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316171    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316174    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316177    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316180    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316182    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316185    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316188    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316191    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316194    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316198    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316202    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316204    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316207    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:29.326296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316210    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316213    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316216    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316219    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316221    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316224    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316226    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316229    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316232    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316234    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316238    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316241    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316243    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316246    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316248    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316251    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316254    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316258    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316261    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:29.326770 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316264    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316267    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316269    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316272    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316275    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316277    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316280    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316283    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316285    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316288    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316291    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316293    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316296    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316299    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316302    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316305    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316307    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316310    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316313    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316316    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:29.327284 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316318    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316321    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316323    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316328    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316330    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316333    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316336    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316340    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316343    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316347    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316352    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316355    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316357    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316360    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316363    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316365    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316368    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316371    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316374    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:29.327856 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316376    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316380    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316383    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316385    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316388    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316390    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316393    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316396    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316398    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316401    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316403    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316406    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316409    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316411    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316414    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316416    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:29.328454 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316420    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.316446    4553 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316485    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316489    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316492    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316495    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316498    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316500    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316505    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316507    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316510    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316514    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316516    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316519    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316521    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316524    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:29.329194 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316527    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316530    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316534    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316538    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316540    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316543    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316546    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316549    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316551    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316554    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316557    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316560    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316562    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316565    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316567    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316570    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316574    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316576    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316579    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:29.329914 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316581    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316584    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316587    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316589    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316592    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316596    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316599    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316602    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316604    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316607    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316609    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316612    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316615    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316617    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316620    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316622    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316625    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316627    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316630    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316632    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:29.330548 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316635    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316637    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316640    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316642    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316646    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316648    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316651    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316654    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316656    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316660    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316663    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316665    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316669    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316673    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316676    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316679    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316682    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316686    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316689    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:29.331049 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316692    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316694    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316697    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316699    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316702    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316704    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316707    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316709    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316712    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316714    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316717    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316719    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316722    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316724    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316726    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316729    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316731    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316734    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316737    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316740    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:29.331648 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316743    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316745    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316750    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316752    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316755    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316757    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316760    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316762    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316765    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316767    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316770    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316772    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316775    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.316777    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.322229    4553 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.322246    4553 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.322266    4553 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.322270    4553 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322330    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322336    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:29.332166 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322339    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322342    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322345    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322348    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322351    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322354    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322356    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322359    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322362    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322365    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322368    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322370    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322373    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322376    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322378    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322382    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322384    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322387    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322389    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:29.332662 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322392    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322394    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322397    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322400    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322402    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322405    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322408    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322410    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322413    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322416    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322418    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322421    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322424    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322426    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322429    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322431    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322434    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322436    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322439    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322441    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:29.333193 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322444    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322446    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322448    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322451    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322454    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322457    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322459    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322462    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322465    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322467    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322470    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322472    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322475    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322478    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322481    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322483    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322486    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322488    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322491    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322493    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:29.333727 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322497    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322501    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322503    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322506    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322509    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322512    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322514    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322517    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322519    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322522    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322524    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322527    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322529    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322532    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322534    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322537    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322540    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322543    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322546    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322549    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:29.334229 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322551    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322554    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322556    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322559    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322562    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322565    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322567    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322570    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322572    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322575    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322577    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322580    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322582    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322585    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322588    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322590    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322593    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322595    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322599    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:29.334733 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322603    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322605    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322608    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322610    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322613    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322616    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.322643    4553 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322796    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322800    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322804    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322807    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322810    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322814    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322817    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322819    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322822    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:29.335448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322825    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322828    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322830    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322833    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322835    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322838    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322840    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322843    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322846    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322850    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322853    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322856    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322859    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322862    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322865    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322868    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322870    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322873    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322876    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:29.336094 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322878    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322881    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322883    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322886    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322888    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322891    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322893    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322896    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322899    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322901    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322905    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322908    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322910    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322913    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322915    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322918    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322921    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322923    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322926    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322928    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:29.336753 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322931    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322933    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322936    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322938    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322941    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322943    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322946    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322949    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322951    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322954    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322956    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322958    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322961    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322963    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322966    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322968    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322971    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322973    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322978    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322981    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:29.337448 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322984    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322986    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322989    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322992    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322994    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322997    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.322999    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323003    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323007    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323010    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323013    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323016    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323019    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323021    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323024    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323027    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323029    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323032    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323034    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:29.337942 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323037    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323039    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323042    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323045    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323047    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323050    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323052    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323055    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323057    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323060    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323063    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323066    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323069    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323072    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323074    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323085    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323089    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323092    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:29.338463 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323094    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:29.338899 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.323119    4553 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:42:29.338899 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323176    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:42:29.338899 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323181    4553 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:42:29.338899 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323184    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:42:29.338899 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323187    4553 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:42:29.338899 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323189    4553 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:42:29.338899 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323192    4553 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:42:29.338899 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323195    4553 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:42:29.338899 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323197    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:42:29.338899 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323200    4553 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:42:29.338899 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323202    4553 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:42:29.338899 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323205    4553 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:42:29.338899 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323208    4553 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:42:29.338899 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323211    4553 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:42:29.338899 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323213    4553 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323216    4553 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323219    4553 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323221    4553 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323223    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323226    4553 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323228    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323231    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323234    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323236    4553 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323238    4553 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323242    4553 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323245    4553 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323248    4553 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323250    4553 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323253    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323256    4553 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323259    4553 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323261    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323264    4553 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:42:29.339366 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323267    4553 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323269    4553 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323272    4553 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323274    4553 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323277    4553 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323279    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323282    4553 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323284    4553 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323287    4553 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323289    4553 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323292    4553 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323294    4553 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323297    4553 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323301    4553 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323304    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323307    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323310    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323313    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323315    4553 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:42:29.339849 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323318    4553 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323320    4553 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323323    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323326    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323330    4553 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323333    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323336    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323338    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323341    4553 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323344    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323347    4553 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323350    4553 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323352    4553 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323355    4553 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323357    4553 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323359    4553 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323362    4553 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323364    4553 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323368    4553 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323370    4553 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:42:29.340322 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323373    4553 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323375    4553 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323378    4553 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323381    4553 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323383    4553 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323386    4553 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323388    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323391    4553 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323393    4553 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323395    4553 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323398    4553 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323400    4553 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323403    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323405    4553 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323408    4553 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323411    4553 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323414    4553 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323417    4553 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323419    4553 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323422    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:42:29.340814 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323424    4553 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323427    4553 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323430    4553 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323432    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323436    4553 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323438    4553 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323441    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323443    4553 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323446    4553 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323448    4553 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323451    4553 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323453    4553 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323458    4553 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:29.323463    4553 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.323534    4553 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.324527    4553 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.324587    4553 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.324765    4553 server.go:1014] "Starting client certificate rotation"
Oct 30 21:42:29.341382 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.324899    4553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:42:29.341859 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.324942    4553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:36:48 +0000 UTC" deadline="2028-06-30 14:40:00.092516839 +0000 UTC"
Oct 30 21:42:29.341859 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.324969    4553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14608h57m30.767549529s"
Oct 30 21:42:29.341859 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.325286    4553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:42:29.341859 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.326455    4553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:42:29.341859 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.328082    4553 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:42:29.341859 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.330063    4553 log.go:25] "Validated CRI v1 image API"
Oct 30 21:42:29.341859 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.330786    4553 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:42:29.341859 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.332388    4553 fs.go:135] Filesystem UUIDs: map[433e1ca6-4f16-46c9-9134-60f05ef43913:/dev/nvme0n1p4 5080a781-e22a-4fe2-9baa-9b8febaee5e0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 21:42:29.342095 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.332409    4553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15e6800bd921632283fb5a0ab9ad47002b7144c9d49e0137d1826f30c14dd4af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15e6800bd921632283fb5a0ab9ad47002b7144c9d49e0137d1826f30c14dd4af/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34b9ef2673106cb292bf45d813a921de92268e9d16f98a1f6a51edb850bd94a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34b9ef2673106cb292bf45d813a921de92268e9d16f98a1f6a51edb850bd94a1/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3baa19f3af396a1f221a972f4506f42e206e135a6cb1a8d09dd35e822303be3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3baa19f3af396a1f221a972f4506f42e206e135a6cb1a8d09dd35e822303be3a/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e0c5f28917c5e12df3379aee4c16573ddaa61395e446cd45321b8d79415d73f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e0c5f28917c5e12df3379aee4c16573ddaa61395e446cd45321b8d79415d73f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/414b472a25b2ce5a0df7e80951a1b5acf023d6c1c217e166ffc9ed3657aabec6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/414b472a25b2ce5a0df7e80951a1b5acf023d6c1c217e166ffc9ed3657aabec6/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49c5bcdd64661713256a5b41de3be06730c40a54c824eb52061aa389a4ec6dd4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49c5bcdd64661713256a5b41de3be06730c40a54c824eb52061aa389a4ec6dd4/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d86251e01020a78a38815518237105966945da8f412b835ec16334fb49fa2ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d86251e01020a78a38815518237105966945da8f412b835ec16334fb49fa2ed/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5fc53b1e96939496d00537dbc80bfc2a02d885734e207857daf817cb2d7643b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5fc53b1e96939496d00537dbc80bfc2a02d885734e207857daf817cb2d7643b3/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9390004a5986cc084c3803f6262f3703bea203c3b145ac3be46ea854f4128d9f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9390004a5986cc084c3803f6262f3703bea203c3b145ac3be46ea854f4128d9f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e203ac76cdf33263fb80e851d1d322a52020e930e8f789f1078c9aa90a1601a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e203ac76cdf33263fb80e851d1d322a52020e930e8f789f1078c9aa90a1601a/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa91d14fe097a8fffacc20e7b97ea5f6c7887ddefd3b1c21348c548657fc392c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa91d14fe097a8fffacc20e7b97ea5f6c7887ddefd3b1c21348c548657fc392c/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4738fd074784866f876219762ab832e8877702a595da98471afaf3b97968b5e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4738fd074784866f876219762ab832e8877702a595da98471afaf3b97968b5e/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c978e4740588cc682044f40fc8939dd3907c788d3e687cd192d8915277d23f0f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c978e4740588cc682044f40fc8939dd3907c788d3e687cd192d8915277d23f0f/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca1e93b98daa51c4a97fba771bf8881d8c845b6006db5a50f711c6a952c067ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca1e93b98daa51c4a97fba771bf8881d8c845b6006db5a50f711c6a952c067ed/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5e9e682eb747437d890f42d5d370b3d3251267f18458346a398f78419834eae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5e9e682eb747437d890f42d5d370b3d3251267f18458346a398f78419834eae/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e316df61719a014965aaec92d5c4c1b1730a803c5dd9d96dafedd8d68ef3e52f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e316df61719a014965aaec92d5c4c1b1730a803c5dd9d96dafedd8d68ef3e52f/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00223ac9-d79b-42d2-ae49-0f17451b2228/volumes/kubernetes.io~projected/kube-api-access-bn64w:{mountpoint:/var/lib/kubelet/pods/00223ac9-d79b-42d2-ae49-0f17451b2228/volumes/kubernetes.io~projected/kube-api-access-bn64w major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04aee07d-cc70-47b8-b101-d2b3b7111198/volumes/kubernetes.io~projected/kube-api-access-bcrtw:{mountpoint:/var/lib/kubelet/pods/04aee07d-cc70-47b8-b101-d2b3b7111198/volumes/kubernetes.io~projected/kube-api-access-bcrtw major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/volumes/kubernetes.io~projected/kube-api-access-hvmtp:{mountpoint:/var/lib/kubelet/pods/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/volumes/kubernetes.io~projected/kube-api-access-hvmtp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3/volumes/kubernetes.io~projected/kube-api-access-4zvtp:{mountpoint:/var/lib/kubelet/pods/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3/volumes/kubernetes.io~projected/kube-api-access-4zvtp major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47734525-f46d-41de-93b8-c8b8f4087651/volumes/kubernetes.io~projected/kube-api-access-2d6fv:{mountpoint:/var/lib/kubelet/pods/47734525-f46d-41de-93b8-c8b8f4087651/volumes/kubernetes.io~projected/kube-api-access-2d6fv major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49253fd8-a9b7-457a-8104-1df48059e145/volumes/kubernetes.io~projected/kube-api-access-l49w6:{mountpoint:/var/lib/kubelet/pods/49253fd8-a9b7-457a-8104-1df48059e145/volumes/kubernetes.io~projected/kube-api-access-l49w6 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49253fd8-a9b7-457a-8104-1df48059e145/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/49253fd8-a9b7-457a-8104-1df48059e145/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f970ab0-3a72-4464-b767-d6867503e81d/volumes/kubernetes.io~projected/kube-api-access-7z876:{mountpoint:/var/lib/kubelet/pods/4f970ab0-3a72-4464-b767-d6867503e81d/volumes/kubernetes.io~projected/kube-api-access-7z876 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2/volumes/kubernetes.io~projected/kube-api-access-jfh9d:{mountpoint:/var/lib/kubelet/pods/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2/volumes/kubernetes.io~projected/kube-api-access-jfh9d major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6eb9603c-7aa1-4d59-afc7-5fc921bf2505/volumes/kubernetes.io~projected/kube-api-access-sbfft:{mountpoint:/var/lib/kubelet/pods/6eb9603c-7aa1-4d59-afc7-5fc921bf2505/volumes/kubernetes.io~projected/kube-api-access-sbfft major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88edf321-028a-4bb5-b729-1359bd54fb57/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/88edf321-028a-4bb5-b729-1359bd54fb57/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/937068c6-0ce1-4485-afc1-0c42a8a16aba/volumes/kubernetes.io~projected/kube-api-access-dlrxc:{mountpoint:/var/lib/kubelet/pods/937068c6-0ce1-4485-afc1-0c42a8a16aba/volumes/kubernetes.io~projected/kube-api-access-dlrxc major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/937068c6-0ce1-4485-afc1-0c42a8a16aba/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/937068c6-0ce1-4485-afc1-0c42a8a16aba/volumes/kubernetes.io~secret/signing-key major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/946b7964-c272-43fc-84f3-2e37be76568f/volumes/kubernetes.io~projected/kube-api-access-sdm95:{mountpoint:/var/lib/kubelet/pods/946b7964-c272-43fc-84f3-2e37be76568f/volumes/kubernetes.io~projected/kube-api-access-sdm95 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a392100e-84e5-4329-8e2c-6257b15fc58a/volumes/kubernetes.io~projected/kube-api-access-cn7pc:{mountpoint:/var/lib/kubelet/pods/a392100e-84e5-4329-8e2c-6257b15fc58a/volumes/kubernetes.io~projected/kube-api-access-cn7pc major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c370a2e7-4648-47ed-ae04-4d6d03ee323e/volumes/kubernetes.io~projected/kube-api-access-25zcv:{mountpoint:/var/lib/kubelet/pods/c370a2e7-4648-47ed-ae04-4d6d03ee323e/volumes/kubernetes.io~projected/kube-api-access-25zcv major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83/volumes/kubernetes.io~projected/kube-api-access-npb5n:{mountpoint:/var/lib/kubelet/pods/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83/volumes/kubernetes.io~projected/kube-api-access-npb5n major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f07039a3-5a8c-4bdf-8241-47942c8d359d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f07039a3-5a8c-4bdf-8241-47942c8d359d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f07039a3-5a8c-4bdf-8241-47942c8d359d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f07039a3-5a8c-4bdf-8241-47942c8d359d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f07039a3-5a8c-4bdf-8241-47942c8d359d/volumes/kubernetes.io~projected/kube-api-access-8kr6b:{mountpoint:/var/lib/kubelet/pods/f07039a3-5a8c-4bdf-8241-47942c8d359d/volumes/kubernetes.io~projected/kube-api-access-8kr6b major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9430e64-a396-496f-a23a-96439631e58a/volumes/kubernetes.io~projected/kube-api-access-ghwzq:{mountpoint:/var/lib/kubelet/pods/f9430e64-a396-496f-a23a-96439631e58a/volumes/kubernetes.io~projected/kube-api-access-ghwzq major:0 minor:67 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ff8c9512914015f477d43aa7c5c16ecbaa39208712783073ecde43d41ed03929/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/99fb9dc29437483025e42bc22457c4bc767abf410e6a1582324256f4f03ef5a2/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/76dbfeec28f1daaf29a329882ba80ac53c7bb5426a77a0ffa2ac24e0f51dfaa7/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/70047bdc7ea06a1b191d3798d73341b79ab9675a5184de04ed1680b73460e7bc/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/5bc047262a852d467e2321c73cc3e0472467757698e05f54fc02787f74b7ac54/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/dccb73880fd5c260d3b703cc8428818474aee50e5e32ff3a52b6e642a8489983/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/97a78aa7d3d60201719ecff4763abf30156a30020e0cd6500f50b826988e1b18/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/379c3c0f913f00269eaa524618a4d4be529b75c023c6352796ee7e7a64debbed/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3b93ba6c591d6bc27074753005e66ede479e5eb7075917e1542f0493bd4c1815/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/f803bcfc053d5a8e455ea46fb95eb86d3ed790148d6f60f212a03f1db6683816/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/7b5c80609851321c59e2682ba9e96d283d4fb4f8874ee24500f215ed2c0303d9/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/7510741d94ad251f0dbd379ee4c90e6bbdb2e3d42bf0d1cd429fd521c54fbb27/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a53039a63245507e159bf637e959619ac6f775aab79e29c0b4fdffff68019fa5/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/fcbde0c0e4973a050e38171d678e5c98da34520b28bb1621707c8e5df934f54a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7161ed5d98370d5fcd61118783debb5828ba8fb83781b16219ef3eee6ed2aaee/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/caf7942b7e960bab25f1d133b815e08229ec76c976a49acb5868484624081181/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/332587e97474aa85a8569db6985e004f4f65fb259d68c7be392cc4c6d13b96c7/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0ac677278e1e9f8c3eb4e04bf829961f3c28d731e21521d11621fd3245f1dd4c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-183:{mountpoint:/var/lib/containers/storage/overlay/5785a5ec02b25c655b9d301f89d4dffd56ec38f3e047b4423c59587e52b3e953/merged major:0 minor:183 fsType:overlay blockSize:0} overlay_0-188:{mountpoint:/var/lib/containers/storage/overlay/17375a72e2ac42053fbe5b365da9ffc91bbab6e9e2d5ed9cb8fdcaf0fadf938c/merged major:0 minor:188 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/dda5f710ac3f7b0f11a147567a7007edadccb929f0ff84bcf6f158dde8ae6065/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/458c9454b7d9969264c6c710c4469e709c158c4ad180ad064a345c06670e3d3f/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/55a1bd896c73cd30e4a0580e02fa9349b366c5b5c3fc223c78930eeb7ea0208f/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/45702d133686df8000da09ad64a1e7b50c37f25d4930b0aaf63d5cdb4aee9784/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/784ad352d93badff5f2913a5bc71f0f174cc7871c46edd5692be36c22f9fbdc9/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/f22a800e90e33c46809cc417235294c049fdbd2a2155b9b1bf867278b76c6d51/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/206bf6f5fad8c93cceba58c44d9c31e08ed49287b4e2eff3df6fd8855f301fd4/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/be8fe12aa54f1520fe5d07c43de13cbae4a457c5cbb0add22c9c29b3a811ff21/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/98dfde4c7ed707694c053f95b287915baf11c3555a75be45849efdd2d31ffdb5/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/5fd4f22b01ece6b9cc063ff3c395ad4b2253d46dd5d33ce6fa4f3d47eba4078a/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/5f52cdf4565e26151addde316e3a9e3357241fc00f508553e3a7b776a4b436bb/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/b32290c30c5f1e1a50d190bbc558cf0616a8d2252630fab20c4913d5aaf18602/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/299f4c41e037c69f5ee262701caf0f6ab2be82b504850aa2d46408c59410e78b/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/50f0eb995bebe832b4db6c8d256ce5b2b589b9ec8a69af07503e15c3b0b17569/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6285f2ecbe843327ce0913a08345466d434d7c3d0fd6ee8a40944c92c4ffdaba/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/231c5687d842cb6c5fe5608f1005b443f99457e339b2c34e5e6bc6ec21201038/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a6a641f8399865d36e61f201ea6e21638add12629d1876dafec9c1c67dad72eb/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/87d0a1aa7fbfa358b9b848165d02525812d1e1476c4bb74da7185c8d4f1d0dd7/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/824dc44f71c88241b0fa98de586177593fa59df3014147d6127fba3c21a33677/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/3211347a61e4bffa4d5f58b21f425a88fae7e4ba5c3542de517469d328ea806b/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/22470b6ecf7880b0de263a1059f64bf1989e561b0b8fbf205b76968e003523e7/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/77b48038b3975b4ac5e609588a084b7449962539b9bca7f05011df9a14b76749/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6307d56eb77fc3346da6c5f52cd014a8c94ccc63c94aa50b3104dd147772c845/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.340461    4553 manager.go:217] Machine: {Timestamp:2026-10-30 21:42:29.339989625 +0000 UTC m=+0.074352654 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100735 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21343e03222336fd4eea6cb9bf09ec SystemUUID:ec21343e-0322-2336-fd4e-ea6cb9bf09ec BootID:695eabdd-9568-43d9-8ae6-692fd594701d Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/6eb9603c-7aa1-4d59-afc7-5fc921bf2505/volumes/kubernetes.io~projected/kube-api-access-sbfft DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-188 DeviceMajor:0 DeviceMinor:188 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/volumes/kubernetes.io~projected/kube-api-access-hvmtp DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/937068c6-0ce1-4485-afc1-0c42a8a16aba/volumes/kubernetes.io~projected/kube-api-access-dlrxc DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5e9e682eb747437d890f42d5d370b3d3251267f18458346a398f78419834eae/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e0c5f28917c5e12df3379aee4c16573ddaa61395e446cd45321b8d79415d73f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49c5bcdd64661713256a5b41de3be06730c40a54c824eb52061aa389a4ec6dd4/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9430e64-a396-496f-a23a-96439631e58a/volumes/kubernetes.io~projected/kube-api-access-ghwzq DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3baa19f3af396a1f221a972f4506f42e206e135a6cb1a8d09dd35e822303be3a/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5fc53b1e96939496d00537dbc80bfc2a02d885734e207857daf817cb2d7643b3/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c978e4740588cc682044f40fc8939dd3907c788d3e687cd192d8915277d23f0f/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f07039a3-5a8c-4bdf-8241-47942c8d359d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a392100e-84e5-4329-8e2c-6257b15fc58a/volumes/kubernetes.io~projected/kube-api-access-cn7pc DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa91d14fe097a8fffacc20e7b97ea5f6c7887ddefd3b1c21348c548657fc392c/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d86251e01020a78a38815518237105966945da8f412b835ec16334fb49fa2ed/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e316df61719a014965aaec92d5c4c1b1730a803c5dd9d96dafedd8d68ef3e52f/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca1e93b98daa51c4a97fba771bf8881d8c845b6006db5a50f711c6a952c067ed/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9390004a5986cc084c3803f6262f3703bea203c3b145ac3be46ea854f4128d9f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-183 DeviceMajor:0 DeviceMinor:183 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/414b472a25b2ce5a0df7e80951a1b5acf023d6c1c217e166ffc9ed3657aabec6/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34b9ef2673106cb292bf45d813a921de92268e9d16f98a1f6a51edb850bd94a1/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49253fd8-a9b7-457a-8104-1df48059e145/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/47734525-f46d-41de-93b8-c8b8f4087651/volumes/kubernetes.io~projected/kube-api-access-2d6fv DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/937068c6-0ce1-4485-afc1-0c42a8a16aba/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88edf321-028a-4bb5-b729-1359bd54fb57/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/49253fd8-a9b7-457a-8104-1df48059e145/volumes/kubernetes.io~projected/kube-api-access-l49w6 DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/00223ac9-d79b-42d2-ae49-0f17451b2228/volumes/kubernetes.io~projected/kube-api-access-bn64w DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c370a2e7-4648-47ed-ae04-4d6d03ee323e/volumes/kubernetes.io~projected/kube-api-access-25zcv DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2/volumes/kubernetes.io~projected/kube-api-access-jfh9d DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f07039a3-5a8c-4bdf-8241-47942c8d359d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f07039a3-5a8c-4bdf-8241-47942c8d359d/volumes/kubernetes.io~projected/kube-api-access-8kr6b DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3/volumes/kubernetes.io~projected/kube-api-access-4zvtp DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e203ac76cdf33263fb80e851d1d322a52020e930e8f789f1078c9aa90a1601a/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/946b7964-c272-43fc-84f3-2e37be76568f/volumes/kubernetes.io~projected/kube-api-access-sdm95 DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83/volumes/kubernetes.io~projected/kube-api-access-npb5n DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4738fd074784866f876219762ab832e8877702a595da98471afaf3b97968b5e/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f970ab0-3a72-4464-b767-d6867503e81d/volumes/kubernetes.io~projected/kube-api-access-7z876 DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/04aee07d-cc70-47b8-b101-d2b3b7111198/volumes/kubernetes.io~projected/kube-api-access-bcrtw DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15e6800bd921632283fb5a0ab9ad47002b7144c9d49e0137d1826f30c14dd4af/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:15e6800bd921632 MacAddress:32:f2:2d:5f:79:08 Speed:10000 Mtu:8901} {Name:49c5bcdd6466171 MacAddress:4e:9f:a2:9a:6f:49 Speed:10000 Mtu:8901} {Name:4d86251e01020a7 MacAddress:06:02:0d:58:12:6b Speed:10000 Mtu:8901} {Name:5fc53b1e9693949 MacAddress:02:9f:d7:7c:04:28 Speed:10000 Mtu:8901} {Name:aa91d14fe097a8f MacAddress:22:1c:74:5b:5b:08 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:19:77:6b:51:33 Speed:0 Mtu:9001} {Name:br-int MacAddress:8e:0e:02:b4:a6:18 Speed:0 Mtu:8901} {Name:c978e4740588cc6 MacAddress:c2:e7:68:17:c5:0e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:19:77:6b:51:33 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ee:26:a0:bf:e1:b3 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:aa:e5:4c:69:a2:7d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.340752    4553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.340802    4553 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.341062    4553 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.341081    4553 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-226.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.341227    4553 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.341234    4553 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.341249    4553 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.341261    4553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.341362    4553 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.341518    4553 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.341627    4553 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.341662    4553 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.341672    4553 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.341681    4553 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.341691    4553 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:42:29.342252 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.341700    4553 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:42:29.342861 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.342370    4553 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:42:29.342861 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.342382    4553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:42:29.342936 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.342910    4553 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:42:29.344947 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.344934    4553 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:42:29.344947 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.344947    4553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:42:29.345066 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345051    4553 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:42:29.345273 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345262    4553 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:42:29.345312 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345292    4553 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:42:29.345411 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345402    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:42:29.345443 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345417    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:42:29.345443 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345423    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:42:29.345443 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345432    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:42:29.345443 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345441    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:42:29.345542 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345450    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:42:29.345542 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345467    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:42:29.345542 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345473    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:42:29.345542 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345480    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:42:29.345542 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345486    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:42:29.345542 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345498    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:42:29.345542 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345507    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:42:29.345542 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345535    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:42:29.345542 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.345544    4553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:42:29.349028 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.349012    4553 server.go:1263] "Started kubelet"
Oct 30 21:42:29.349127 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.349066    4553 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:42:29.354163 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.349190    4553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:42:29.354163 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.349266    4553 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:42:29.354163 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.349921    4553 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:42:29.354163 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.352353    4553 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:42:29.349568 ip-10-0-133-226 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:42:29.355098 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.354336    4553 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:42:29.355098 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.354353    4553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:42:29.355098 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.354484    4553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:36:50 +0000 UTC" deadline="2028-04-05 10:53:50.346069708 +0000 UTC"
Oct 30 21:42:29.355098 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.354518    4553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12541h11m20.991554451s"
Oct 30 21:42:29.355098 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.354521    4553 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:42:29.355098 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.354505    4553 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:42:29.355098 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.354552    4553 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:42:29.355098 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:29.355044    4553 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:42:29.356732 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356686    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47734525-f46d-41de-93b8-c8b8f4087651" volumeName="kubernetes.io/projected/47734525-f46d-41de-93b8-c8b8f4087651-kube-api-access-2d6fv" seLinuxMountContext=""
Oct 30 21:42:29.356732 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356731    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f970ab0-3a72-4464-b767-d6867503e81d" volumeName="kubernetes.io/projected/4f970ab0-3a72-4464-b767-d6867503e81d-kube-api-access-7z876" seLinuxMountContext=""
Oct 30 21:42:29.356903 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356752    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2" volumeName="kubernetes.io/configmap/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:42:29.356903 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356766    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00223ac9-d79b-42d2-ae49-0f17451b2228" volumeName="kubernetes.io/projected/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-api-access-bn64w" seLinuxMountContext=""
Oct 30 21:42:29.356903 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356784    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d0ce829-dc72-4f8e-91ea-2a5313fa7b11" volumeName="kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:42:29.356903 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356796    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d0ce829-dc72-4f8e-91ea-2a5313fa7b11" volumeName="kubernetes.io/secret/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:42:29.356903 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356809    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49253fd8-a9b7-457a-8104-1df48059e145" volumeName="kubernetes.io/projected/49253fd8-a9b7-457a-8104-1df48059e145-kube-api-access-l49w6" seLinuxMountContext=""
Oct 30 21:42:29.356903 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356824    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d1fccd5-6eaa-4a43-9a1e-d67c1005e369" volumeName="kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret" seLinuxMountContext=""
Oct 30 21:42:29.356903 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356840    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f970ab0-3a72-4464-b767-d6867503e81d" volumeName="kubernetes.io/configmap/4f970ab0-3a72-4464-b767-d6867503e81d-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:42:29.356903 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356857    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04aee07d-cc70-47b8-b101-d2b3b7111198" volumeName="kubernetes.io/configmap/04aee07d-cc70-47b8-b101-d2b3b7111198-serviceca" seLinuxMountContext=""
Oct 30 21:42:29.356903 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356870    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47734525-f46d-41de-93b8-c8b8f4087651" volumeName="kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-service-ca-bundle" seLinuxMountContext=""
Oct 30 21:42:29.356903 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356888    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2" volumeName="kubernetes.io/configmap/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:42:29.356903 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356900    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2" volumeName="kubernetes.io/projected/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-kube-api-access-jfh9d" seLinuxMountContext=""
Oct 30 21:42:29.357454 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356912    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" volumeName="kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:42:29.357454 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356934    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="946b7964-c272-43fc-84f3-2e37be76568f" volumeName="kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95" seLinuxMountContext=""
Oct 30 21:42:29.357454 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356948    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f07039a3-5a8c-4bdf-8241-47942c8d359d" volumeName="kubernetes.io/projected/f07039a3-5a8c-4bdf-8241-47942c8d359d-kube-api-access-8kr6b" seLinuxMountContext=""
Oct 30 21:42:29.357454 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356964    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9430e64-a396-496f-a23a-96439631e58a" volumeName="kubernetes.io/projected/f9430e64-a396-496f-a23a-96439631e58a-kube-api-access-ghwzq" seLinuxMountContext=""
Oct 30 21:42:29.357454 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356978    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00223ac9-d79b-42d2-ae49-0f17451b2228" volumeName="kubernetes.io/empty-dir/00223ac9-d79b-42d2-ae49-0f17451b2228-data-volume" seLinuxMountContext=""
Oct 30 21:42:29.357454 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.356995    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d0ce829-dc72-4f8e-91ea-2a5313fa7b11" volumeName="kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-env-overrides" seLinuxMountContext=""
Oct 30 21:42:29.357454 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357007    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3" volumeName="kubernetes.io/projected/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-kube-api-access-4zvtp" seLinuxMountContext=""
Oct 30 21:42:29.357454 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357022    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="937068c6-0ce1-4485-afc1-0c42a8a16aba" volumeName="kubernetes.io/configmap/937068c6-0ce1-4485-afc1-0c42a8a16aba-signing-cabundle" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357486    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="937068c6-0ce1-4485-afc1-0c42a8a16aba" volumeName="kubernetes.io/projected/937068c6-0ce1-4485-afc1-0c42a8a16aba-kube-api-access-dlrxc" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357506    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="937068c6-0ce1-4485-afc1-0c42a8a16aba" volumeName="kubernetes.io/secret/937068c6-0ce1-4485-afc1-0c42a8a16aba-signing-key" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357525    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a392100e-84e5-4329-8e2c-6257b15fc58a" volumeName="kubernetes.io/configmap/a392100e-84e5-4329-8e2c-6257b15fc58a-config-volume" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357538    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a392100e-84e5-4329-8e2c-6257b15fc58a" volumeName="kubernetes.io/projected/a392100e-84e5-4329-8e2c-6257b15fc58a-kube-api-access-cn7pc" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357555    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47734525-f46d-41de-93b8-c8b8f4087651" volumeName="kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357569    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47734525-f46d-41de-93b8-c8b8f4087651" volumeName="kubernetes.io/empty-dir/47734525-f46d-41de-93b8-c8b8f4087651-archives-path" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357583    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a392100e-84e5-4329-8e2c-6257b15fc58a" volumeName="kubernetes.io/empty-dir/a392100e-84e5-4329-8e2c-6257b15fc58a-tmp-dir" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357608    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c370a2e7-4648-47ed-ae04-4d6d03ee323e" volumeName="kubernetes.io/projected/c370a2e7-4648-47ed-ae04-4d6d03ee323e-kube-api-access-25zcv" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357625    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f07039a3-5a8c-4bdf-8241-47942c8d359d" volumeName="kubernetes.io/empty-dir/f07039a3-5a8c-4bdf-8241-47942c8d359d-tmp" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357639    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9430e64-a396-496f-a23a-96439631e58a" volumeName="kubernetes.io/empty-dir/f9430e64-a396-496f-a23a-96439631e58a-tmp-dir" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357651    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" volumeName="kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357669    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" volumeName="kubernetes.io/projected/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-kube-api-access-sbfft" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357682    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88edf321-028a-4bb5-b729-1359bd54fb57" volumeName="kubernetes.io/configmap/88edf321-028a-4bb5-b729-1359bd54fb57-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357700    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d0ce829-dc72-4f8e-91ea-2a5313fa7b11" volumeName="kubernetes.io/projected/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-kube-api-access-hvmtp" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357713    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49253fd8-a9b7-457a-8104-1df48059e145" volumeName="kubernetes.io/secret/49253fd8-a9b7-457a-8104-1df48059e145-cert" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357727    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" volumeName="kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357743    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88edf321-028a-4bb5-b729-1359bd54fb57" volumeName="kubernetes.io/secret/88edf321-028a-4bb5-b729-1359bd54fb57-agent-certs" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357756    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04aee07d-cc70-47b8-b101-d2b3b7111198" volumeName="kubernetes.io/projected/04aee07d-cc70-47b8-b101-d2b3b7111198-kube-api-access-bcrtw" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357773    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d0ce829-dc72-4f8e-91ea-2a5313fa7b11" volumeName="kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovnkube-config" seLinuxMountContext=""
Oct 30 21:42:29.357822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357785    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f07039a3-5a8c-4bdf-8241-47942c8d359d" volumeName="kubernetes.io/empty-dir/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-tuned" seLinuxMountContext=""
Oct 30 21:42:29.358476 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357900    4553 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2f6dd07-b2ef-43b9-af3a-9bf016a45c83" volumeName="kubernetes.io/projected/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-kube-api-access-npb5n" seLinuxMountContext=""
Oct 30 21:42:29.358476 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357940    4553 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:42:29.358476 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.357949    4553 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:42:29.358965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.358943    4553 factory.go:55] Registering systemd factory
Oct 30 21:42:29.358965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.358965    4553 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:42:29.359343 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.359326    4553 factory.go:153] Registering CRI-O factory
Oct 30 21:42:29.359390 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.359349    4553 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:42:29.359420 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.359408    4553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:42:29.359455 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.359438    4553 factory.go:103] Registering Raw factory
Oct 30 21:42:29.359493 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.359458    4553 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:42:29.359808 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.359790    4553 manager.go:319] Starting recovery of all containers
Oct 30 21:42:29.375060 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.375033    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 21:42:29.375386 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.375365    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 21:42:29.383017 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.382978    4553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:42:29.384883 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.384865    4553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:42:29.384996 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.384983    4553 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:42:29.385106 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.385010    4553 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:42:29.385106 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:29.385051    4553 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:42:29.391826 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.391799    4553 generic.go:356] "Generic (PLEG): container finished" podID="c0e7e76dbf05397217f3bc5792d5aaba" containerID="ee810f4b1c126e5ce65ff914844ac4e8c14584120f12ab8ec15de1ed4f237a27" exitCode=0
Oct 30 21:42:29.403855 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.403815    4553 generic.go:356] "Generic (PLEG): container finished" podID="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" containerID="3146e2d8e579405d73043ac31f3364939393c5a8ca973d63ac39d42a16453f38" exitCode=0
Oct 30 21:42:29.403855 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.403850    4553 generic.go:356] "Generic (PLEG): container finished" podID="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" containerID="669a8406837fe61416e66bffcc660c46e4cd77a6e3618a6d335f8eaca598e6e5" exitCode=0
Oct 30 21:42:29.403855 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.403862    4553 generic.go:356] "Generic (PLEG): container finished" podID="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" containerID="20d42ee4018891e916d3e823da839a9d83e98648c7e148c2bce904edd5933969" exitCode=0
Oct 30 21:42:29.404124 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.403870    4553 generic.go:356] "Generic (PLEG): container finished" podID="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" containerID="e52f208291178a3a17a945fa6429abf0777381c818237959048327e9803987bd" exitCode=0
Oct 30 21:42:29.404124 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.403876    4553 generic.go:356] "Generic (PLEG): container finished" podID="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" containerID="e8f4c2720fd286999e50150d4d46da9b4906901f45a43c9092ee7ac79e42e9a9" exitCode=0
Oct 30 21:42:29.404124 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.403886    4553 generic.go:356] "Generic (PLEG): container finished" podID="6eb9603c-7aa1-4d59-afc7-5fc921bf2505" containerID="13f8e566eb7194db4aa6d831def238a4360fc765a7d84b520a2bcac1f41532c4" exitCode=0
Oct 30 21:42:29.424876 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.424846    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 21:42:29.425360 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.425331    4553 generic.go:356] "Generic (PLEG): container finished" podID="0d0ce829-dc72-4f8e-91ea-2a5313fa7b11" containerID="cc45f1ac17a281a416834f923e19f5286a63595ef14bd4ac0fe424aecb98a133" exitCode=1
Oct 30 21:42:29.434526 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.434508    4553 manager.go:324] Recovery completed
Oct 30 21:42:29.447333 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.447280    4553 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:29.450199 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.450178    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:29.450348 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.450328    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:29.450425 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.450359    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:29.452115 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452097    4553 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:42:29.452115 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452109    4553 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:29.452115 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452114    4553 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:42:29.452285 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452157    4553 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:42:29.452317 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452306    4553 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:42:29.452345 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452320    4553 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:42:29.452345 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452343    4553 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:42:29.452402 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452352    4553 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:42:29.452402 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452359    4553 policy_none.go:50] "Start"
Oct 30 21:42:29.452402 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452370    4553 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:42:29.452402 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452382    4553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:42:29.452563 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452551    4553 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:42:29.452591 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452566    4553 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:42:29.452591 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452574    4553 policy_none.go:44] "Start"
Oct 30 21:42:29.452938 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452921    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:29.452938 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452949    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:29.453064 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.452961    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:29.455312 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:29.455295    4553 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:42:29.457180 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.457129    4553 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:42:29.460694 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.460677    4553 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:42:29.460776 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:29.460705    4553 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:42:29.460776 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.460716    4553 server.go:87] "Starting device plugin registration server"
Oct 30 21:42:29.461023 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.461008    4553 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:42:29.461091 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.461022    4553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:42:29.461163 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.461100    4553 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:42:29.461283 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.461244    4553 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:42:29.461348 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.461298    4553 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:42:29.461348 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.461306    4553 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:42:29.461441 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.461353    4553 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:42:29.461288547Z","UUID":"d7d894f1-7497-434a-bc41-2feea323ce31","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:42:29.462741 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:29.462712    4553 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:42:29.462828 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:29.462753    4553 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-226.us-west-1.compute.internal\" not found"
Oct 30 21:42:29.462965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.462947    4553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:42:29.463030 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.462974    4553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:42:29.486652 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.486043    4553 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"]
Oct 30 21:42:29.486837 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.486752    4553 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:29.487837 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.487817    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:29.487946 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.487856    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:29.487946 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.487872    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:29.488210 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.488195    4553 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:29.488346 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.488329    4553 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:29.489182 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.489163    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:29.489276 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.489198    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:29.489276 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.489214    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:29.489276 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.489219    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:29.489276 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.489245    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:29.489276 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.489255    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:29.489419 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:29.489377    4553 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found" node="ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.489526 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.489495    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal" event={"ID":"c0e7e76dbf05397217f3bc5792d5aaba","Type":"ContainerStarted","Data":"414b472a25b2ce5a0df7e80951a1b5acf023d6c1c217e166ffc9ed3657aabec6"}
Oct 30 21:42:29.489574 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.489563    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal" event={"ID":"24d6104f1ec87a54e6dc2f912238418d","Type":"ContainerStarted","Data":"a4eaf106c0d079f3543f72fd272d8e88254e2280fdd438178a9d420deb84cbe7"}
Oct 30 21:42:29.489603 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.489579    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal" event={"ID":"24d6104f1ec87a54e6dc2f912238418d","Type":"ContainerStarted","Data":"d5e9e682eb747437d890f42d5d370b3d3251267f18458346a398f78419834eae"}
Oct 30 21:42:29.489636 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.489602    4553 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:29.490381 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.490364    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:29.490444 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.490391    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:29.490444 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.490408    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:29.490549 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:29.490537    4553 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-226.us-west-1.compute.internal\" not found" node="ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.554652 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.554617    4553 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:42:29.554829 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.554682    4553 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:42:29.554829 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.554716    4553 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:42:29.554904 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.554897    4553 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:42:29.555972 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.555960    4553 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.562121 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.562107    4553 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:42:29.563245 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.563230    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:42:29.563306 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.563261    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:42:29.563306 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.563273    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:42:29.563306 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.563307    4553 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.605379 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.605346    4553 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:42:29.654248 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.654216    4553 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:42:29.654383 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:29.654272    4553 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.654383 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.654293    4553 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.655724 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.655599    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c0e7e76dbf05397217f3bc5792d5aaba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"c0e7e76dbf05397217f3bc5792d5aaba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.655892 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.655749    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0e7e76dbf05397217f3bc5792d5aaba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"c0e7e76dbf05397217f3bc5792d5aaba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.655892 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.655789    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/24d6104f1ec87a54e6dc2f912238418d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"24d6104f1ec87a54e6dc2f912238418d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.707017 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.706934    4553 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.707017 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.707014    4553 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.707719 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.707699    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeNotReady"
Oct 30 21:42:29.707779 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.707735    4553 setters.go:546] "Node became not ready" node="ip-10-0-133-226.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T21:42:29Z","lastTransitionTime":"2026-10-30T21:42:29Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 21:42:29.711393 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.711378    4553 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:42:29.711466 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:29.711419    4553 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.721404 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.721375    4553 nodeinfomanager.go:471] existing CSINode "ip-10-0-133-226.us-west-1.compute.internal" is owned by different node (oldNodeID="939689d6-0b50-4938-986f-c66bafd2b1e3", newNodeID=""), cleaning up...
Oct 30 21:42:29.756665 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.756631    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0e7e76dbf05397217f3bc5792d5aaba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"c0e7e76dbf05397217f3bc5792d5aaba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.756801 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.756674    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/24d6104f1ec87a54e6dc2f912238418d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"24d6104f1ec87a54e6dc2f912238418d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.756801 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.756700    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c0e7e76dbf05397217f3bc5792d5aaba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"c0e7e76dbf05397217f3bc5792d5aaba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.756801 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.756751    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c0e7e76dbf05397217f3bc5792d5aaba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"c0e7e76dbf05397217f3bc5792d5aaba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.756801 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.756759    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/24d6104f1ec87a54e6dc2f912238418d-config\") pod \"kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"24d6104f1ec87a54e6dc2f912238418d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:29.756801 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:29.756752    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0e7e76dbf05397217f3bc5792d5aaba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal\" (UID: \"c0e7e76dbf05397217f3bc5792d5aaba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:30.342116 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.342081    4553 apiserver.go:52] "Watching apiserver"
Oct 30 21:42:30.432720 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.432686    4553 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:30.432862 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.432812    4553 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:30.455072 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.455044    4553 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:42:30.455602 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.455579    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-7d5lt","openshift-service-ca/service-ca-5fffc97bdf-pbghc","kube-system/konnectivity-agent-prfrp","kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw","openshift-multus/network-metrics-daemon-8kpvh","openshift-multus/multus-additional-cni-plugins-l25db","openshift-ovn-kubernetes/ovnkube-node-s8pzx","kube-system/global-pull-secret-syncer-x7hcg","openshift-cluster-node-tuning-operator/tuned-ch6rj","openshift-insights/insights-runtime-extractor-nx7wb","openshift-dns/dns-default-x6dnl","openshift-kube-storage-version-migrator/migrator-66df5d496b-p2t8s","openshift-ingress-canary/ingress-canary-9tvgl","openshift-image-registry/node-ca-rr82t","openshift-network-operator/iptables-alerter-4wg8g","openshift-insights/periodic-gathering-fpxlc-s8s8f","openshift-multus/multus-9g8ct","openshift-dns/node-resolver-9hg74"]
Oct 30 21:42:30.457590 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.457567    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:30.460600 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.460580    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:30.464097 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.464044    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:30.494249 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.494219    4553 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:42:30.494371 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:30.494276    4553 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:30.494856 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.494843    4553 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:42:30.494888 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:30.494878    4553 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-226.us-west-1.compute.internal"
Oct 30 21:42:30.510284 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.510251    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:42:30.510284 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.510261    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:30.510284 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.510260    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:30.511065 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.511047    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p7xlk\""
Oct 30 21:42:30.511219 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.511164    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7tj7w\""
Oct 30 21:42:30.511219 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.511178    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9csh8\""
Oct 30 21:42:30.511219 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.511169    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6gscd\""
Oct 30 21:42:30.511219 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.511213    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vtv5j\""
Oct 30 21:42:30.511596 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.511581    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:42:30.534616 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.534582    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:42:30.534616 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.534609    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:42:30.534855 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.534687    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:42:30.534855 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.534749    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:42:30.534855 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.534768    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:42:30.534855 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.534782    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:42:30.534855 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.534796    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:42:30.534855 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.534805    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:42:30.534855 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.534819    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:42:30.534855 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.534852    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:42:30.535167 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.534919    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:42:30.535167 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.535001    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:42:30.536925 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.536910    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:42:30.537025 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.536963    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:42:30.537025 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.536986    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:42:30.537368 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537345    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:42:30.537368 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537350    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:42:30.537482 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537404    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:42:30.537482 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537474    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:42:30.537630 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537613    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 21:42:30.537684 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537675    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:42:30.537824 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537682    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:42:30.537824 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537684    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:42:30.537824 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537724    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:42:30.537824 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537766    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:42:30.537824 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537775    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:42:30.538034 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537892    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:42:30.538034 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537911    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 21:42:30.538034 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537933    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:42:30.538034 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537960    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:30.538034 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.537997    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 21:42:30.544682 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.544665    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:42:30.544730 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.544702    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:42:30.555972 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.555939    4553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:42:30.560007 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.559989    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-etc-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.560084 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560018    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovnkube-script-lib\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.560084 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560035    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdm95\" (UniqueName: \"kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95\") pod \"network-check-target-7d5lt\" (UID: \"946b7964-c272-43fc-84f3-2e37be76568f\") " pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:30.560084 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560056    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.560246 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560126    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zvtp\" (UniqueName: \"kubernetes.io/projected/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-kube-api-access-4zvtp\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.560246 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560177    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-cnibin\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.560246 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560203    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-hostroot\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.560246 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560225    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a392100e-84e5-4329-8e2c-6257b15fc58a-config-volume\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:30.560392 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560248    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49253fd8-a9b7-457a-8104-1df48059e145-cert\") pod \"ingress-canary-9tvgl\" (UID: \"49253fd8-a9b7-457a-8104-1df48059e145\") " pod="openshift-ingress-canary/ingress-canary-9tvgl"
Oct 30 21:42:30.560392 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560292    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:30.560392 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560333    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9430e64-a396-496f-a23a-96439631e58a-tmp-dir\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:42:30.560392 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560356    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f970ab0-3a72-4464-b767-d6867503e81d-iptables-alerter-script\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:42:30.560576 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560431    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04aee07d-cc70-47b8-b101-d2b3b7111198-serviceca\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:42:30.560576 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560443    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9430e64-a396-496f-a23a-96439631e58a-tmp-dir\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:42:30.560576 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560502    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-system-cni-dir\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.560576 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560526    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-log-socket\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.560576 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560538    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a392100e-84e5-4329-8e2c-6257b15fc58a-config-volume\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:30.560576 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560550    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn64w\" (UniqueName: \"kubernetes.io/projected/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-api-access-bn64w\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:30.560576 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560565    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:42:30.560904 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560653    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-multus-certs\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.560904 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560679    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/04aee07d-cc70-47b8-b101-d2b3b7111198-serviceca\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:42:30.560904 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560737    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4f970ab0-3a72-4464-b767-d6867503e81d-iptables-alerter-script\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:42:30.560904 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560765    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-etc-kubernetes\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.560904 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560820    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysconfig\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.560904 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560857    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysctl-conf\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.560904 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560887    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-run-netns\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.561123 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560921    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-cni-bin\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.561123 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560959    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-registration-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.561123 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.560989    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/00223ac9-d79b-42d2-ae49-0f17451b2228-data-volume\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:30.561123 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561014    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9430e64-a396-496f-a23a-96439631e58a-hosts-file\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:42:30.561123 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561071    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-daemon-config\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.561123 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561083    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/00223ac9-d79b-42d2-ae49-0f17451b2228-data-volume\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:30.561123 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561112    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfh9d\" (UniqueName: \"kubernetes.io/projected/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-kube-api-access-jfh9d\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.561512 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561153    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcrtw\" (UniqueName: \"kubernetes.io/projected/04aee07d-cc70-47b8-b101-d2b3b7111198-kube-api-access-bcrtw\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:42:30.561512 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561180    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-kubelet-config\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:30.561512 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561210    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-ovn\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.561512 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561238    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l49w6\" (UniqueName: \"kubernetes.io/projected/49253fd8-a9b7-457a-8104-1df48059e145-kube-api-access-l49w6\") pod \"ingress-canary-9tvgl\" (UID: \"49253fd8-a9b7-457a-8104-1df48059e145\") " pod="openshift-ingress-canary/ingress-canary-9tvgl"
Oct 30 21:42:30.561512 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561401    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-cni-binary-copy\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.561512 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561442    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-kubelet\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.561512 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561471    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-daemon-config\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.561512 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561476    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04aee07d-cc70-47b8-b101-d2b3b7111198-host\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:42:30.561952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561558    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a392100e-84e5-4329-8e2c-6257b15fc58a-tmp-dir\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:30.561952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561593    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn7pc\" (UniqueName: \"kubernetes.io/projected/a392100e-84e5-4329-8e2c-6257b15fc58a-kube-api-access-cn7pc\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:30.561952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561621    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-slash\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.561952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561633    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a392100e-84e5-4329-8e2c-6257b15fc58a-tmp-dir\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:30.561952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561645    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-device-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.561952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561637    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-cni-binary-copy\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.561952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561682    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-cni-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.561952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561729    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/47734525-f46d-41de-93b8-c8b8f4087651-archives-path\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:30.561952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561788    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-service-ca-bundle\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:30.561952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561834    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/47734525-f46d-41de-93b8-c8b8f4087651-archives-path\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:30.561952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561840    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-run\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.561952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561878    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88edf321-028a-4bb5-b729-1359bd54fb57-konnectivity-ca\") pod \"konnectivity-agent-prfrp\" (UID: \"88edf321-028a-4bb5-b729-1359bd54fb57\") " pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:42:30.561952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561928    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-dbus\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:30.561952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561935    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-service-ca-bundle\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:30.561952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561965    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/937068c6-0ce1-4485-afc1-0c42a8a16aba-signing-key\") pod \"service-ca-5fffc97bdf-pbghc\" (UID: \"937068c6-0ce1-4485-afc1-0c42a8a16aba\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.561992    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrxc\" (UniqueName: \"kubernetes.io/projected/937068c6-0ce1-4485-afc1-0c42a8a16aba-kube-api-access-dlrxc\") pod \"service-ca-5fffc97bdf-pbghc\" (UID: \"937068c6-0ce1-4485-afc1-0c42a8a16aba\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562023    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovn-node-metrics-cert\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562056    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/00223ac9-d79b-42d2-ae49-0f17451b2228-crio-socket\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562088    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-conf-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562124    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-modprobe-d\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562186    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562197    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/88edf321-028a-4bb5-b729-1359bd54fb57-konnectivity-ca\") pod \"konnectivity-agent-prfrp\" (UID: \"88edf321-028a-4bb5-b729-1359bd54fb57\") " pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562217    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-sys-fs\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562267    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-cni-multus\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562328    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d6fv\" (UniqueName: \"kubernetes.io/projected/47734525-f46d-41de-93b8-c8b8f4087651-kube-api-access-2d6fv\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562374    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562438    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-node-log\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562458    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovnkube-config\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562488    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmtp\" (UniqueName: \"kubernetes.io/projected/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-kube-api-access-hvmtp\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562519    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-socket-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562583    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-kubernetes\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562617    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-lib-modules\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562651    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kr6b\" (UniqueName: \"kubernetes.io/projected/f07039a3-5a8c-4bdf-8241-47942c8d359d-kube-api-access-8kr6b\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562710    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.562738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562747    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovnkube-config\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562759    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-cni-netd\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562792    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562818    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npb5n\" (UniqueName: \"kubernetes.io/projected/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-kube-api-access-npb5n\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562843    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-systemd\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562896    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-var-lib-kubelet\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562923    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-host\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562943    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cnibin\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.562967    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563019    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-cni-bin\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563053    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysctl-d\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563092    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cni-binary-copy\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563119    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/937068c6-0ce1-4485-afc1-0c42a8a16aba-signing-cabundle\") pod \"service-ca-5fffc97bdf-pbghc\" (UID: \"937068c6-0ce1-4485-afc1-0c42a8a16aba\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563159    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-var-lib-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563193    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563216    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563292    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-system-cni-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563314    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cni-binary-copy\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563325    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-socket-dir-parent\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563353    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/937068c6-0ce1-4485-afc1-0c42a8a16aba-signing-cabundle\") pod \"service-ca-5fffc97bdf-pbghc\" (UID: \"937068c6-0ce1-4485-afc1-0c42a8a16aba\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563375    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f970ab0-3a72-4464-b767-d6867503e81d-host-slash\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563404    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563445    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563462    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-k8s-cni-cncf-io\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563477    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-systemd\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563490    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563506    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-netns\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563550    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-tuned\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563609    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-systemd-units\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563668    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghwzq\" (UniqueName: \"kubernetes.io/projected/f9430e64-a396-496f-a23a-96439631e58a-kube-api-access-ghwzq\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563675    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-tuned\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.563742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563697    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-sys\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.564754 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563789    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfft\" (UniqueName: \"kubernetes.io/projected/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-kube-api-access-sbfft\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.564754 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563832    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25zcv\" (UniqueName: \"kubernetes.io/projected/c370a2e7-4648-47ed-ae04-4d6d03ee323e-kube-api-access-25zcv\") pod \"migrator-66df5d496b-p2t8s\" (UID: \"c370a2e7-4648-47ed-ae04-4d6d03ee323e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p2t8s"
Oct 30 21:42:30.564754 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563865    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-kubelet\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.564754 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563899    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.564754 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.563968    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-env-overrides\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.564754 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.564005    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07039a3-5a8c-4bdf-8241-47942c8d359d-tmp\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.564754 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.564072    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07039a3-5a8c-4bdf-8241-47942c8d359d-tmp\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.564754 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.564090    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88edf321-028a-4bb5-b729-1359bd54fb57-agent-certs\") pod \"konnectivity-agent-prfrp\" (UID: \"88edf321-028a-4bb5-b729-1359bd54fb57\") " pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:42:30.564754 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.564115    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.564754 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.564170    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:30.564754 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.564167    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-env-overrides\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.564754 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.564197    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z876\" (UniqueName: \"kubernetes.io/projected/4f970ab0-3a72-4464-b767-d6867503e81d-kube-api-access-7z876\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:42:30.564754 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.564214    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-trusted-ca-bundle\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:30.570490 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.570471    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovnkube-script-lib\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.585298 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.585266    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:42:30.586113 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.586095    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:42:30.594874 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.594854    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-trusted-ca-bundle\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:30.611642 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.611615    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:42:30.611764 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.611705    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xlldj\""
Oct 30 21:42:30.611819 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.611773    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 21:42:30.611819 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.611801    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nrkss\""
Oct 30 21:42:30.611874 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.611827    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pkpl9\""
Oct 30 21:42:30.611874 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.611829    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:42:30.611949 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.611873    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-82rrg\""
Oct 30 21:42:30.612204 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612092    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sflz7\""
Oct 30 21:42:30.612204 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612160    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wcr9d\""
Oct 30 21:42:30.612204 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612198    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j7s9f\""
Oct 30 21:42:30.612396 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612202    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:42:30.612396 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612215    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zffrj\""
Oct 30 21:42:30.612396 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612269    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-ovn-node-metrics-cert\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.612396 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612306    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:42:30.612396 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612391    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5fg22\""
Oct 30 21:42:30.612599 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612427    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7xfbb\""
Oct 30 21:42:30.612599 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612518    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:42:30.612599 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612524    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/937068c6-0ce1-4485-afc1-0c42a8a16aba-signing-key\") pod \"service-ca-5fffc97bdf-pbghc\" (UID: \"937068c6-0ce1-4485-afc1-0c42a8a16aba\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:30.612599 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612516    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9qhlv\""
Oct 30 21:42:30.612599 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612574    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:42:30.612766 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612601    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dzgdj\""
Oct 30 21:42:30.612766 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612672    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-msnq6\""
Oct 30 21:42:30.612766 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.612713    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:42:30.615301 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.615283    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/88edf321-028a-4bb5-b729-1359bd54fb57-agent-certs\") pod \"konnectivity-agent-prfrp\" (UID: \"88edf321-028a-4bb5-b729-1359bd54fb57\") " pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:42:30.617467 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.617446    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zvtp\" (UniqueName: \"kubernetes.io/projected/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-kube-api-access-4zvtp\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.617467 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.617450    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdm95\" (UniqueName: \"kubernetes.io/projected/946b7964-c272-43fc-84f3-2e37be76568f-kube-api-access-sdm95\") pod \"network-check-target-7d5lt\" (UID: \"946b7964-c272-43fc-84f3-2e37be76568f\") " pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:30.619394 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.619376    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcrtw\" (UniqueName: \"kubernetes.io/projected/04aee07d-cc70-47b8-b101-d2b3b7111198-kube-api-access-bcrtw\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:42:30.619691 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.619677    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfh9d\" (UniqueName: \"kubernetes.io/projected/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-kube-api-access-jfh9d\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.620153 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.620113    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn64w\" (UniqueName: \"kubernetes.io/projected/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-api-access-bn64w\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:30.620321 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.620304    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d6fv\" (UniqueName: \"kubernetes.io/projected/47734525-f46d-41de-93b8-c8b8f4087651-kube-api-access-2d6fv\") pod \"periodic-gathering-fpxlc-s8s8f\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") " pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:42:30.620507 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.620489    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49253fd8-a9b7-457a-8104-1df48059e145-cert\") pod \"ingress-canary-9tvgl\" (UID: \"49253fd8-a9b7-457a-8104-1df48059e145\") " pod="openshift-ingress-canary/ingress-canary-9tvgl"
Oct 30 21:42:30.621444 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.620973    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kr6b\" (UniqueName: \"kubernetes.io/projected/f07039a3-5a8c-4bdf-8241-47942c8d359d-kube-api-access-8kr6b\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.621444 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.621107    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbfft\" (UniqueName: \"kubernetes.io/projected/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-kube-api-access-sbfft\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.621444 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.621123    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn7pc\" (UniqueName: \"kubernetes.io/projected/a392100e-84e5-4329-8e2c-6257b15fc58a-kube-api-access-cn7pc\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:30.621444 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.621269    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrxc\" (UniqueName: \"kubernetes.io/projected/937068c6-0ce1-4485-afc1-0c42a8a16aba-kube-api-access-dlrxc\") pod \"service-ca-5fffc97bdf-pbghc\" (UID: \"937068c6-0ce1-4485-afc1-0c42a8a16aba\") " pod="openshift-service-ca/service-ca-5fffc97bdf-pbghc"
Oct 30 21:42:30.621444 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.621315    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghwzq\" (UniqueName: \"kubernetes.io/projected/f9430e64-a396-496f-a23a-96439631e58a-kube-api-access-ghwzq\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:42:30.621444 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.621407    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25zcv\" (UniqueName: \"kubernetes.io/projected/c370a2e7-4648-47ed-ae04-4d6d03ee323e-kube-api-access-25zcv\") pod \"migrator-66df5d496b-p2t8s\" (UID: \"c370a2e7-4648-47ed-ae04-4d6d03ee323e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-p2t8s"
Oct 30 21:42:30.621444 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.621424    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l49w6\" (UniqueName: \"kubernetes.io/projected/49253fd8-a9b7-457a-8104-1df48059e145-kube-api-access-l49w6\") pod \"ingress-canary-9tvgl\" (UID: \"49253fd8-a9b7-457a-8104-1df48059e145\") " pod="openshift-ingress-canary/ingress-canary-9tvgl"
Oct 30 21:42:30.621728 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.621564    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npb5n\" (UniqueName: \"kubernetes.io/projected/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-kube-api-access-npb5n\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:30.621728 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.621616    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvmtp\" (UniqueName: \"kubernetes.io/projected/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-kube-api-access-hvmtp\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.621728 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.621651    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z876\" (UniqueName: \"kubernetes.io/projected/4f970ab0-3a72-4464-b767-d6867503e81d-kube-api-access-7z876\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:42:30.622608 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.622590    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-original-pull-secret\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:30.665420 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665377    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-ovn\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.665420 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665416    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-kubelet\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.665647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665433    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04aee07d-cc70-47b8-b101-d2b3b7111198-host\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:42:30.665647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665478    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/04aee07d-cc70-47b8-b101-d2b3b7111198-host\") pod \"node-ca-rr82t\" (UID: \"04aee07d-cc70-47b8-b101-d2b3b7111198\") " pod="openshift-image-registry/node-ca-rr82t"
Oct 30 21:42:30.665647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665498    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-ovn\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.665647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665498    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-kubelet\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.665647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665563    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-slash\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.665647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665639    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-slash\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665661    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-device-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665691    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-cni-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665728    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-run\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665750    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-device-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665753    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-dbus\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665780    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/00223ac9-d79b-42d2-ae49-0f17451b2228-crio-socket\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665792    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-cni-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665805    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-conf-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665815    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-run\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665827    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-modprobe-d\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665837    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/00223ac9-d79b-42d2-ae49-0f17451b2228-crio-socket\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665845    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-conf-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665843    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-dbus\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665912    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-sys-fs\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665963    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-sys-fs\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.665965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665963    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-cni-multus\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.665991    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-node-log\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666006    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-cni-multus\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666011    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-socket-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666034    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-kubernetes\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666057    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-lib-modules\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666063    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-node-log\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666071    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-cni-netd\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666077    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-modprobe-d\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666083    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-socket-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666088    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666114    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-kubernetes\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666159    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666117    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-systemd\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666190    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-systemd\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666197    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-var-lib-kubelet\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666202    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-lib-modules\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666213    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-host\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666234    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cnibin\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666258    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-cni-bin\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666273    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysctl-d\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666291    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-var-lib-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666294    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-host\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666298    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-var-lib-kubelet\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666156    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-cni-netd\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666310    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666348    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-system-cni-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666398    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-var-lib-cni-bin\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666417    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-socket-dir-parent\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666416    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-var-lib-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666441    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f970ab0-3a72-4464-b767-d6867503e81d-host-slash\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666443    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-system-cni-dir\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666443    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666445    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-cnibin\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666474    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-multus-socket-dir-parent\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666474    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f970ab0-3a72-4464-b767-d6867503e81d-host-slash\") pod \"iptables-alerter-4wg8g\" (UID: \"4f970ab0-3a72-4464-b767-d6867503e81d\") " pod="openshift-network-operator/iptables-alerter-4wg8g"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666480    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666534    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysctl-d\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666535    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666572    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-k8s-cni-cncf-io\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666593    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-systemd\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666597    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666608    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666624    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-k8s-cni-cncf-io\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666631    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-netns\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666657    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-systemd-units\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666659    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-run-systemd\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666681    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-sys\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:30.666695    4553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666704    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-netns\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666708    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-kubelet\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666741    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666748    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-kubelet\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666743    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-systemd-units\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:30.666770    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs podName:d2f6dd07-b2ef-43b9-af3a-9bf016a45c83 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.166751132 +0000 UTC m=+1.901114141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs") pod "network-metrics-daemon-8kpvh" (UID: "d2f6dd07-b2ef-43b9-af3a-9bf016a45c83") : secret "metrics-daemon-secret" not found
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666771    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-sys\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666778    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-run-ovn-kubernetes\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.666748 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666803    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666830    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666852    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666856    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-etc-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666891    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-etc-openvswitch\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:30.666907    4553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666917    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666940    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-cnibin\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666957    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-hostroot\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:30.666982    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.166965071 +0000 UTC m=+1.901328102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666988    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-hostroot\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666979    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667017    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667020    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-cnibin\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667046    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-system-cni-dir\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667063    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-log-socket\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667089    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-multus-certs\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.666987    4553 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667105    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-log-socket\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667112    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6eb9603c-7aa1-4d59-afc7-5fc921bf2505-system-cni-dir\") pod \"multus-additional-cni-plugins-l25db\" (UID: \"6eb9603c-7aa1-4d59-afc7-5fc921bf2505\") " pod="openshift-multus/multus-additional-cni-plugins-l25db"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667115    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-etc-kubernetes\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:30.667155    4553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667173    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-etc-kubernetes\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667175    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysconfig\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667156    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2-host-run-multus-certs\") pod \"multus-9g8ct\" (UID: \"6bd5141f-18f5-4f4d-a4d2-5f1e170a68e2\") " pod="openshift-multus/multus-9g8ct"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:30.667202    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:31.167188109 +0000 UTC m=+1.901551145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667222    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysconfig\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667233    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysctl-conf\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667250    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-run-netns\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667275    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-cni-bin\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667303    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-run-netns\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667300    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-registration-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667327    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d0ce829-dc72-4f8e-91ea-2a5313fa7b11-host-cni-bin\") pod \"ovnkube-node-s8pzx\" (UID: \"0d0ce829-dc72-4f8e-91ea-2a5313fa7b11\") " pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667344    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9430e64-a396-496f-a23a-96439631e58a-hosts-file\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667335    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f07039a3-5a8c-4bdf-8241-47942c8d359d-etc-sysctl-conf\") pod \"tuned-ch6rj\" (UID: \"f07039a3-5a8c-4bdf-8241-47942c8d359d\") " pod="openshift-cluster-node-tuning-operator/tuned-ch6rj"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667375    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f9430e64-a396-496f-a23a-96439631e58a-hosts-file\") pod \"node-resolver-9hg74\" (UID: \"f9430e64-a396-496f-a23a-96439631e58a\") " pod="openshift-dns/node-resolver-9hg74"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667376    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-kubelet-config\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667434    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d1fccd5-6eaa-4a43-9a1e-d67c1005e369-kubelet-config\") pod \"global-pull-secret-syncer-x7hcg\" (UID: \"4d1fccd5-6eaa-4a43-9a1e-d67c1005e369\") " pod="kube-system/global-pull-secret-syncer-x7hcg"
Oct 30 21:42:30.668516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.667363    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3-registration-dir\") pod \"aws-ebs-csi-driver-node-jx5gw\" (UID: \"3d0fbfbe-62ce-4e9c-96b9-066d7bf1bfd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jx5gw"
Oct 30 21:42:30.669554 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.669477    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a392100e-84e5-4329-8e2c-6257b15fc58a-metrics-tls\") pod \"dns-default-x6dnl\" (UID: \"a392100e-84e5-4329-8e2c-6257b15fc58a\") " pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:30.764780 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:30.764690    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:31.161787 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:31.161759    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x6dnl"]
Oct 30 21:42:31.164818 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:31.164778    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda392100e_84e5_4329_8e2c_6257b15fc58a.slice/crio-eea44f10ff794b3c99143ee83b2a429781f7dbf2d981a7b93bd321d804e0ff77 WatchSource:0}: Error finding container eea44f10ff794b3c99143ee83b2a429781f7dbf2d981a7b93bd321d804e0ff77: Status 404 returned error can't find the container with id eea44f10ff794b3c99143ee83b2a429781f7dbf2d981a7b93bd321d804e0ff77
Oct 30 21:42:31.167053 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:31.167038    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:42:31.171203 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:31.171181    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:31.171277 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:31.171235    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:31.171277 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:31.171258    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:31.171337 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:31.171316    4553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:31.171337 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:31.171322    4553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:31.171405 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:31.171362    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:32.171349803 +0000 UTC m=+2.905712813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:31.171405 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:31.171375    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:32.171369116 +0000 UTC m=+2.905732125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:31.171405 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:31.171381    4553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:42:31.171495 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:31.171430    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs podName:d2f6dd07-b2ef-43b9-af3a-9bf016a45c83 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:32.171418072 +0000 UTC m=+2.905781096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs") pod "network-metrics-daemon-8kpvh" (UID: "d2f6dd07-b2ef-43b9-af3a-9bf016a45c83") : secret "metrics-daemon-secret" not found
Oct 30 21:42:31.436849 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:31.436765    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6dnl" event={"ID":"a392100e-84e5-4329-8e2c-6257b15fc58a","Type":"ContainerStarted","Data":"eea44f10ff794b3c99143ee83b2a429781f7dbf2d981a7b93bd321d804e0ff77"}
Oct 30 21:42:32.179306 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:32.179273    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:32.179519 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:32.179317    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:32.179519 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:32.179338    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:32.179519 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:32.179441    4553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:32.179519 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:32.179450    4553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:42:32.179519 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:32.179502    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.179486622 +0000 UTC m=+4.913849630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:32.179727 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:32.179585    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:34.179571726 +0000 UTC m=+4.913934735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : configmap "kube-rbac-proxy" not found
Oct 30 21:42:32.182353 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:32.182323    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2f6dd07-b2ef-43b9-af3a-9bf016a45c83-metrics-certs\") pod \"network-metrics-daemon-8kpvh\" (UID: \"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83\") " pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:32.258025 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:32.257991    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpvh"
Oct 30 21:42:32.646734 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:32.646698    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8kpvh"]
Oct 30 21:42:32.650487 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:32.650449    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f6dd07_b2ef_43b9_af3a_9bf016a45c83.slice/crio-842ff1d4bf3358d55c439142c059fe3009aab15a1f4a91b9a48ba1a79545ed28 WatchSource:0}: Error finding container 842ff1d4bf3358d55c439142c059fe3009aab15a1f4a91b9a48ba1a79545ed28: Status 404 returned error can't find the container with id 842ff1d4bf3358d55c439142c059fe3009aab15a1f4a91b9a48ba1a79545ed28
Oct 30 21:42:33.445998 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:33.445966    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8kpvh" event={"ID":"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83","Type":"ContainerStarted","Data":"842ff1d4bf3358d55c439142c059fe3009aab15a1f4a91b9a48ba1a79545ed28"}
Oct 30 21:42:33.783587 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:33.783553    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:34.134100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:34.134070    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:34.195919 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:34.195889    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:34.195919 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:34.195925    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:34.196155 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:34.196037    4553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:34.196155 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:34.196089    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls podName:00223ac9-d79b-42d2-ae49-0f17451b2228 nodeName:}" failed. No retries permitted until 2026-10-30 21:42:38.196076098 +0000 UTC m=+8.930439120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nx7wb" (UID: "00223ac9-d79b-42d2-ae49-0f17451b2228") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:42:34.196566 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:34.196542    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/00223ac9-d79b-42d2-ae49-0f17451b2228-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:34.451005 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:34.450963    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6dnl" event={"ID":"a392100e-84e5-4329-8e2c-6257b15fc58a","Type":"ContainerStarted","Data":"8b7872eb290822bfa9a60df111afd92f995e7436ab5256a56be819e67042abdd"}
Oct 30 21:42:34.451204 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:34.451019    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6dnl" event={"ID":"a392100e-84e5-4329-8e2c-6257b15fc58a","Type":"ContainerStarted","Data":"556347cbf1f7949d70ed48e26abdf021525c01cf941320cca69b3c718a3c06ad"}
Oct 30 21:42:34.451204 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:34.451047    4553 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:42:34.451578 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:34.451564    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:35.455835 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:35.455794    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8kpvh" event={"ID":"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83","Type":"ContainerStarted","Data":"e9c5360b2a384a491fa79376de5515d9732872710e52c7846d67c61788d8a7db"}
Oct 30 21:42:35.456251 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:35.455853    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8kpvh" event={"ID":"d2f6dd07-b2ef-43b9-af3a-9bf016a45c83","Type":"ContainerStarted","Data":"70f75dae10012697808dd2385a09f0a3eb146b0eee3c256b24122351c5dd3f41"}
Oct 30 21:42:38.225306 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:38.225266    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:38.227752 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:38.227728    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/00223ac9-d79b-42d2-ae49-0f17451b2228-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nx7wb\" (UID: \"00223ac9-d79b-42d2-ae49-0f17451b2228\") " pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:38.261752 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:38.261716    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nx7wb"
Oct 30 21:42:38.678628 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:38.678597    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nx7wb"]
Oct 30 21:42:38.681703 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:38.681676    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00223ac9_d79b_42d2_ae49_0f17451b2228.slice/crio-ff319d701d176678c94d7363fa6495e1b50466e0f7bccd516ba3c529cdb407cd WatchSource:0}: Error finding container ff319d701d176678c94d7363fa6495e1b50466e0f7bccd516ba3c529cdb407cd: Status 404 returned error can't find the container with id ff319d701d176678c94d7363fa6495e1b50466e0f7bccd516ba3c529cdb407cd
Oct 30 21:42:38.901455 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:38.901427    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-prfrp"
Oct 30 21:42:39.278195 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:39.278163    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s8pzx"
Oct 30 21:42:39.475577 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:39.475521    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nx7wb" event={"ID":"00223ac9-d79b-42d2-ae49-0f17451b2228","Type":"ContainerStarted","Data":"bb6145c72c73d27858f40bc12b4294528e261757599ec35006961f5e2f6ce7c1"}
Oct 30 21:42:39.475761 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:39.475641    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nx7wb" event={"ID":"00223ac9-d79b-42d2-ae49-0f17451b2228","Type":"ContainerStarted","Data":"ff319d701d176678c94d7363fa6495e1b50466e0f7bccd516ba3c529cdb407cd"}
Oct 30 21:42:39.657590 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:39.657559    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:39.954830 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:39.954757    4553 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-226.us-west-1.compute.internal" event="NodeReady"
Oct 30 21:42:40.480538 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:40.480504    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nx7wb" event={"ID":"00223ac9-d79b-42d2-ae49-0f17451b2228","Type":"ContainerStarted","Data":"f5972bdc1b663ec1915d11ef1bc22ecd452ced3d5a2e8a32bee99c429deb0032"}
Oct 30 21:42:42.491922 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:42.491886    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nx7wb" event={"ID":"00223ac9-d79b-42d2-ae49-0f17451b2228","Type":"ContainerStarted","Data":"d725735324ff9f2605271caf04d0198db1ac63756c6c8453d53eecdd1aca3201"}
Oct 30 21:42:44.598516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.598485    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"]
Oct 30 21:42:44.600521 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.600497    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"
Oct 30 21:42:44.671254 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.671222    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7gms\" (UniqueName: \"kubernetes.io/projected/4e62e696-0ac6-43ec-bdf5-d24edc3e06d3-kube-api-access-w7gms\") pod \"prometheus-operator-9cd89b74c-kdqlc\" (UID: \"4e62e696-0ac6-43ec-bdf5-d24edc3e06d3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"
Oct 30 21:42:44.671421 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.671262    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e62e696-0ac6-43ec-bdf5-d24edc3e06d3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kdqlc\" (UID: \"4e62e696-0ac6-43ec-bdf5-d24edc3e06d3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"
Oct 30 21:42:44.671421 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.671284    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e62e696-0ac6-43ec-bdf5-d24edc3e06d3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kdqlc\" (UID: \"4e62e696-0ac6-43ec-bdf5-d24edc3e06d3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"
Oct 30 21:42:44.671421 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.671403    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e62e696-0ac6-43ec-bdf5-d24edc3e06d3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kdqlc\" (UID: \"4e62e696-0ac6-43ec-bdf5-d24edc3e06d3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"
Oct 30 21:42:44.703649 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.703616    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:42:44.703777 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.703656    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 21:42:44.703777 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.703724    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qskgx\""
Oct 30 21:42:44.709387 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.709374    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:42:44.750001 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.749967    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 21:42:44.750001 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.749980    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:42:44.754771 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.754749    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"]
Oct 30 21:42:44.772720 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.772691    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7gms\" (UniqueName: \"kubernetes.io/projected/4e62e696-0ac6-43ec-bdf5-d24edc3e06d3-kube-api-access-w7gms\") pod \"prometheus-operator-9cd89b74c-kdqlc\" (UID: \"4e62e696-0ac6-43ec-bdf5-d24edc3e06d3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"
Oct 30 21:42:44.772789 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.772731    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e62e696-0ac6-43ec-bdf5-d24edc3e06d3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kdqlc\" (UID: \"4e62e696-0ac6-43ec-bdf5-d24edc3e06d3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"
Oct 30 21:42:44.772789 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.772759    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e62e696-0ac6-43ec-bdf5-d24edc3e06d3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kdqlc\" (UID: \"4e62e696-0ac6-43ec-bdf5-d24edc3e06d3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"
Oct 30 21:42:44.772873 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.772790    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e62e696-0ac6-43ec-bdf5-d24edc3e06d3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kdqlc\" (UID: \"4e62e696-0ac6-43ec-bdf5-d24edc3e06d3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"
Oct 30 21:42:44.773551 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.773523    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e62e696-0ac6-43ec-bdf5-d24edc3e06d3-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-kdqlc\" (UID: \"4e62e696-0ac6-43ec-bdf5-d24edc3e06d3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"
Oct 30 21:42:44.775451 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.775423    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4e62e696-0ac6-43ec-bdf5-d24edc3e06d3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-kdqlc\" (UID: \"4e62e696-0ac6-43ec-bdf5-d24edc3e06d3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"
Oct 30 21:42:44.775538 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.775493    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4e62e696-0ac6-43ec-bdf5-d24edc3e06d3-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-kdqlc\" (UID: \"4e62e696-0ac6-43ec-bdf5-d24edc3e06d3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"
Oct 30 21:42:44.830403 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.830366    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7gms\" (UniqueName: \"kubernetes.io/projected/4e62e696-0ac6-43ec-bdf5-d24edc3e06d3-kube-api-access-w7gms\") pod \"prometheus-operator-9cd89b74c-kdqlc\" (UID: \"4e62e696-0ac6-43ec-bdf5-d24edc3e06d3\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"
Oct 30 21:42:44.911089 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:44.911006    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"
Oct 30 21:42:45.276881 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:45.276803    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc"]
Oct 30 21:42:45.279650 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:45.279611    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e62e696_0ac6_43ec_bdf5_d24edc3e06d3.slice/crio-d92884304fc7533df0c9ef62594139fba9edbd2350c1dc2b682973da7a3fad1f WatchSource:0}: Error finding container d92884304fc7533df0c9ef62594139fba9edbd2350c1dc2b682973da7a3fad1f: Status 404 returned error can't find the container with id d92884304fc7533df0c9ef62594139fba9edbd2350c1dc2b682973da7a3fad1f
Oct 30 21:42:45.462088 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:45.462054    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x6dnl"
Oct 30 21:42:45.501503 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:45.501472    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc" event={"ID":"4e62e696-0ac6-43ec-bdf5-d24edc3e06d3","Type":"ContainerStarted","Data":"d92884304fc7533df0c9ef62594139fba9edbd2350c1dc2b682973da7a3fad1f"}
Oct 30 21:42:47.509514 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:47.509479    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc" event={"ID":"4e62e696-0ac6-43ec-bdf5-d24edc3e06d3","Type":"ContainerStarted","Data":"371b6796e1c6b5cc6d43829f08a020025628f4925705aa00556a818d95a3c6e6"}
Oct 30 21:42:47.509908 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:47.509531    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc" event={"ID":"4e62e696-0ac6-43ec-bdf5-d24edc3e06d3","Type":"ContainerStarted","Data":"abbe12c7265d332b0b146287a2bd3b8ae4d96578c935406605445179728b5444"}
Oct 30 21:42:47.629832 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:47.629777    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-kdqlc" podStartSLOduration=1.937275873 podStartE2EDuration="3.629762527s" totalImagesPullingTime="1.692486654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:47.629226926 +0000 UTC m=+18.363589957" watchObservedRunningTime="2026-10-30 21:42:47.629762527 +0000 UTC m=+18.364125607"
Oct 30 21:42:51.236370 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.236333    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"]
Oct 30 21:42:51.238387 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.238370    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"
Oct 30 21:42:51.315943 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.315910    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c58cf9e5-68a2-4fd7-a364-d7a82686ee5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rcbl7\" (UID: \"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"
Oct 30 21:42:51.315943 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.315959    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c58cf9e5-68a2-4fd7-a364-d7a82686ee5d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rcbl7\" (UID: \"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"
Oct 30 21:42:51.316180 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.316081    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c58cf9e5-68a2-4fd7-a364-d7a82686ee5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rcbl7\" (UID: \"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"
Oct 30 21:42:51.316180 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.316111    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxm8\" (UniqueName: \"kubernetes.io/projected/c58cf9e5-68a2-4fd7-a364-d7a82686ee5d-kube-api-access-hzxm8\") pod \"openshift-state-metrics-7bd7489957-rcbl7\" (UID: \"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"
Oct 30 21:42:51.341976 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.341928    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:42:51.341976 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.341931    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 21:42:51.342163 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.341931    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xll8d\""
Oct 30 21:42:51.342648 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.342631    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"]
Oct 30 21:42:51.393253 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.393224    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"]
Oct 30 21:42:51.395731 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.395710    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sbg2m"]
Oct 30 21:42:51.395850 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.395832    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.397796 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.397774    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.416832 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.416811    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c58cf9e5-68a2-4fd7-a364-d7a82686ee5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rcbl7\" (UID: \"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"
Oct 30 21:42:51.416956 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.416840    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxm8\" (UniqueName: \"kubernetes.io/projected/c58cf9e5-68a2-4fd7-a364-d7a82686ee5d-kube-api-access-hzxm8\") pod \"openshift-state-metrics-7bd7489957-rcbl7\" (UID: \"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"
Oct 30 21:42:51.416956 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.416864    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c58cf9e5-68a2-4fd7-a364-d7a82686ee5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rcbl7\" (UID: \"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"
Oct 30 21:42:51.417062 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.416996    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c58cf9e5-68a2-4fd7-a364-d7a82686ee5d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rcbl7\" (UID: \"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"
Oct 30 21:42:51.417597 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.417538    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c58cf9e5-68a2-4fd7-a364-d7a82686ee5d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-rcbl7\" (UID: \"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"
Oct 30 21:42:51.419366 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.419348    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c58cf9e5-68a2-4fd7-a364-d7a82686ee5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-rcbl7\" (UID: \"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"
Oct 30 21:42:51.419445 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.419358    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c58cf9e5-68a2-4fd7-a364-d7a82686ee5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-rcbl7\" (UID: \"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"
Oct 30 21:42:51.452391 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.452351    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"]
Oct 30 21:42:51.474820 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.474789    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxm8\" (UniqueName: \"kubernetes.io/projected/c58cf9e5-68a2-4fd7-a364-d7a82686ee5d-kube-api-access-hzxm8\") pod \"openshift-state-metrics-7bd7489957-rcbl7\" (UID: \"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"
Oct 30 21:42:51.496536 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.496465    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-csjdz\""
Oct 30 21:42:51.496536 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.496470    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:42:51.496536 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.496491    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 21:42:51.496536 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.496517    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:42:51.496763 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.496559    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:42:51.496763 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.496637    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zk7kt\""
Oct 30 21:42:51.496763 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.496678    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 21:42:51.496763 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.496702    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:42:51.518314 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.518286    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62254772-614a-4bba-a726-6866cd578123-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.518469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.518332    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ffe6dba-999b-4dec-ac96-20124d51b0f7-metrics-client-ca\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.518469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.518364    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw96z\" (UniqueName: \"kubernetes.io/projected/62254772-614a-4bba-a726-6866cd578123-kube-api-access-qw96z\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.518469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.518418    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/62254772-614a-4bba-a726-6866cd578123-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.518630 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.518523    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ffe6dba-999b-4dec-ac96-20124d51b0f7-sys\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.518630 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.518602    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ffe6dba-999b-4dec-ac96-20124d51b0f7-root\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.518734 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.518635    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ffe6dba-999b-4dec-ac96-20124d51b0f7-node-exporter-tls\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.518734 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.518666    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/62254772-614a-4bba-a726-6866cd578123-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.518734 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.518709    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84sx4\" (UniqueName: \"kubernetes.io/projected/8ffe6dba-999b-4dec-ac96-20124d51b0f7-kube-api-access-84sx4\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.518882 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.518766    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/62254772-614a-4bba-a726-6866cd578123-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.518882 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.518821    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ffe6dba-999b-4dec-ac96-20124d51b0f7-node-exporter-textfile\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.518882 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.518847    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ffe6dba-999b-4dec-ac96-20124d51b0f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.518996 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.518879    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62254772-614a-4bba-a726-6866cd578123-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.518996 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.518917    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ffe6dba-999b-4dec-ac96-20124d51b0f7-node-exporter-wtmp\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.518996 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.518950    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ffe6dba-999b-4dec-ac96-20124d51b0f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.550721 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.550695    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"
Oct 30 21:42:51.620318 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.620285    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/62254772-614a-4bba-a726-6866cd578123-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.620457 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.620333    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ffe6dba-999b-4dec-ac96-20124d51b0f7-sys\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.620457 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.620363    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ffe6dba-999b-4dec-ac96-20124d51b0f7-root\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.620457 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.620428    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ffe6dba-999b-4dec-ac96-20124d51b0f7-root\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.620555 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.620546    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ffe6dba-999b-4dec-ac96-20124d51b0f7-sys\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.620584 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.620572    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ffe6dba-999b-4dec-ac96-20124d51b0f7-node-exporter-tls\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.620949 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.620612    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/62254772-614a-4bba-a726-6866cd578123-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.620949 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.620747    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84sx4\" (UniqueName: \"kubernetes.io/projected/8ffe6dba-999b-4dec-ac96-20124d51b0f7-kube-api-access-84sx4\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.620949 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.620797    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/62254772-614a-4bba-a726-6866cd578123-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.620949 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.620830    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ffe6dba-999b-4dec-ac96-20124d51b0f7-node-exporter-textfile\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.620949 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.620854    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ffe6dba-999b-4dec-ac96-20124d51b0f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.620949 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.620890    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62254772-614a-4bba-a726-6866cd578123-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.620949 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.620916    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ffe6dba-999b-4dec-ac96-20124d51b0f7-node-exporter-wtmp\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.620949 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.620938    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ffe6dba-999b-4dec-ac96-20124d51b0f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.622109 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.621355    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/62254772-614a-4bba-a726-6866cd578123-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.622109 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.621422    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62254772-614a-4bba-a726-6866cd578123-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.622109 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.621492    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ffe6dba-999b-4dec-ac96-20124d51b0f7-node-exporter-wtmp\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.622109 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.621529    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ffe6dba-999b-4dec-ac96-20124d51b0f7-metrics-client-ca\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.622109 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.621560    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw96z\" (UniqueName: \"kubernetes.io/projected/62254772-614a-4bba-a726-6866cd578123-kube-api-access-qw96z\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.622109 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.621696    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ffe6dba-999b-4dec-ac96-20124d51b0f7-node-exporter-accelerators-collector-config\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.622415 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.622319    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62254772-614a-4bba-a726-6866cd578123-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.622635 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.622613    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/62254772-614a-4bba-a726-6866cd578123-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.622702 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.622679    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ffe6dba-999b-4dec-ac96-20124d51b0f7-metrics-client-ca\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.622831 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.622811    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ffe6dba-999b-4dec-ac96-20124d51b0f7-node-exporter-textfile\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.624695 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.624652    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ffe6dba-999b-4dec-ac96-20124d51b0f7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.625117 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.625096    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/62254772-614a-4bba-a726-6866cd578123-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.625475 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.625454    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62254772-614a-4bba-a726-6866cd578123-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.626216 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.626198    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ffe6dba-999b-4dec-ac96-20124d51b0f7-node-exporter-tls\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.681907 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.681873    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw96z\" (UniqueName: \"kubernetes.io/projected/62254772-614a-4bba-a726-6866cd578123-kube-api-access-qw96z\") pod \"kube-state-metrics-5dc8d7d89b-vjssf\" (UID: \"62254772-614a-4bba-a726-6866cd578123\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.682171 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.682150    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84sx4\" (UniqueName: \"kubernetes.io/projected/8ffe6dba-999b-4dec-ac96-20124d51b0f7-kube-api-access-84sx4\") pod \"node-exporter-sbg2m\" (UID: \"8ffe6dba-999b-4dec-ac96-20124d51b0f7\") " pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.709249 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.709212    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"
Oct 30 21:42:51.714714 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.714697    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sbg2m"
Oct 30 21:42:51.723992 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:51.723967    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffe6dba_999b_4dec_ac96_20124d51b0f7.slice/crio-04f18110eb32106a814e3dd19977764af46e199cdac1ecac55215c1b4f4145ea WatchSource:0}: Error finding container 04f18110eb32106a814e3dd19977764af46e199cdac1ecac55215c1b4f4145ea: Status 404 returned error can't find the container with id 04f18110eb32106a814e3dd19977764af46e199cdac1ecac55215c1b4f4145ea
Oct 30 21:42:51.974791 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:51.974764    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7"]
Oct 30 21:42:51.977296 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:51.977270    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58cf9e5_68a2_4fd7_a364_d7a82686ee5d.slice/crio-eb863f0d395f858d9df64fdc1cd9a06b4ff4b8cfd78d24e8dff8aa7442cb241c WatchSource:0}: Error finding container eb863f0d395f858d9df64fdc1cd9a06b4ff4b8cfd78d24e8dff8aa7442cb241c: Status 404 returned error can't find the container with id eb863f0d395f858d9df64fdc1cd9a06b4ff4b8cfd78d24e8dff8aa7442cb241c
Oct 30 21:42:52.086117 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:52.086092    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf"]
Oct 30 21:42:52.090637 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:52.090550    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62254772_614a_4bba_a726_6866cd578123.slice/crio-3e7fe01651f8b944e4838dd0679ccf6bcfd396b971f93abfc32d15e1458e5de3 WatchSource:0}: Error finding container 3e7fe01651f8b944e4838dd0679ccf6bcfd396b971f93abfc32d15e1458e5de3: Status 404 returned error can't find the container with id 3e7fe01651f8b944e4838dd0679ccf6bcfd396b971f93abfc32d15e1458e5de3
Oct 30 21:42:52.530716 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:52.530679    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf" event={"ID":"62254772-614a-4bba-a726-6866cd578123","Type":"ContainerStarted","Data":"3e7fe01651f8b944e4838dd0679ccf6bcfd396b971f93abfc32d15e1458e5de3"}
Oct 30 21:42:52.532549 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:52.532523    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7" event={"ID":"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d","Type":"ContainerStarted","Data":"810a7ea5675f69798d4f68c7b7e17e4da80291de729c6a046243c3e877637416"}
Oct 30 21:42:52.532660 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:52.532578    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7" event={"ID":"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d","Type":"ContainerStarted","Data":"d90fdce07a74f5fb9c7b6620a013c027ca92d2d129d6d232c9c3723e715d88c6"}
Oct 30 21:42:52.532660 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:52.532598    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7" event={"ID":"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d","Type":"ContainerStarted","Data":"eb863f0d395f858d9df64fdc1cd9a06b4ff4b8cfd78d24e8dff8aa7442cb241c"}
Oct 30 21:42:52.533628 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:52.533608    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbg2m" event={"ID":"8ffe6dba-999b-4dec-ac96-20124d51b0f7","Type":"ContainerStarted","Data":"04f18110eb32106a814e3dd19977764af46e199cdac1ecac55215c1b4f4145ea"}
Oct 30 21:42:52.998014 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:52.997979    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-75fdf7cdb5-fwp6s"]
Oct 30 21:42:53.009591 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.009568    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.066315 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.065835    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:42:53.066315 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.065887    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fpvt6\""
Oct 30 21:42:53.066315 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.065999    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:42:53.066315 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.066091    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:42:53.066315 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.066187    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:42:53.066315 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.066201    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:42:53.066315 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.066242    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:42:53.066315 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.066292    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:42:53.066820 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.066351    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75fdf7cdb5-fwp6s"]
Oct 30 21:42:53.117032 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.116993    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:42:53.134072 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.134030    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-service-ca\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.134278 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.134081    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db9f14c4-8934-45ff-9330-4f57121b1464-tmp\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.134278 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.134203    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db9f14c4-8934-45ff-9330-4f57121b1464-console-serving-cert\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.134278 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.134252    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8jpk\" (UniqueName: \"kubernetes.io/projected/db9f14c4-8934-45ff-9330-4f57121b1464-kube-api-access-z8jpk\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.134467 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.134286    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-oauth-serving-cert\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.134467 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.134368    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-trusted-ca-bundle\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.134467 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.134405    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-console-config\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.134467 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.134436    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db9f14c4-8934-45ff-9330-4f57121b1464-console-oauth-config\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.235542 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.235503    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-trusted-ca-bundle\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.235779 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.235555    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-console-config\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.235779 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.235585    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db9f14c4-8934-45ff-9330-4f57121b1464-console-oauth-config\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.235779 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.235623    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-service-ca\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.235779 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.235645    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db9f14c4-8934-45ff-9330-4f57121b1464-tmp\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.235779 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.235674    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db9f14c4-8934-45ff-9330-4f57121b1464-console-serving-cert\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.235779 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.235702    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8jpk\" (UniqueName: \"kubernetes.io/projected/db9f14c4-8934-45ff-9330-4f57121b1464-kube-api-access-z8jpk\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.235779 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.235730    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-oauth-serving-cert\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.236496 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.236472    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-oauth-serving-cert\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.237317 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.237290    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-trusted-ca-bundle\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.237670 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.237643    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db9f14c4-8934-45ff-9330-4f57121b1464-tmp\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.237784 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.237761    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-service-ca\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.237849 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.237832    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-console-config\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.240069 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.240050    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db9f14c4-8934-45ff-9330-4f57121b1464-console-oauth-config\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.240583 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.240560    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db9f14c4-8934-45ff-9330-4f57121b1464-console-serving-cert\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.301422 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.301333    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8jpk\" (UniqueName: \"kubernetes.io/projected/db9f14c4-8934-45ff-9330-4f57121b1464-kube-api-access-z8jpk\") pod \"console-75fdf7cdb5-fwp6s\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") " pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.333478 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.333307    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:42:53.733219 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.733186    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75fdf7cdb5-fwp6s"]
Oct 30 21:42:53.761389 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:53.761355    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9f14c4_8934_45ff_9330_4f57121b1464.slice/crio-4ac7b71e1e1035c04166f8c1f5375f3c81ad806689c8a1ae5fd308e14d530e1a WatchSource:0}: Error finding container 4ac7b71e1e1035c04166f8c1f5375f3c81ad806689c8a1ae5fd308e14d530e1a: Status 404 returned error can't find the container with id 4ac7b71e1e1035c04166f8c1f5375f3c81ad806689c8a1ae5fd308e14d530e1a
Oct 30 21:42:53.897442 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.897405    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"]
Oct 30 21:42:53.909956 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.909931    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:53.966681 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.966644    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 21:42:53.966888 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.966810    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 21:42:53.966888 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.966869    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 21:42:53.967032 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.966952    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 21:42:53.967032 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.966956    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ddlxh\""
Oct 30 21:42:53.967032 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.967015    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 21:42:53.967558 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.967525    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-br865l2s7hu0i\""
Oct 30 21:42:53.967849 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:53.967829    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"]
Oct 30 21:42:54.043241 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.043155    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-tmp\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.043241 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.043199    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-metrics-client-ca\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.043241 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.043219    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.043488 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.043272    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.043488 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.043313    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.043488 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.043340    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvzzl\" (UniqueName: \"kubernetes.io/projected/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-kube-api-access-dvzzl\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.043488 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.043363    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-thanos-querier-tls\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.043488 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.043442    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-grpc-tls\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.043488 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.043472    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.144227 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.144188    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-grpc-tls\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.144408 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.144239    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.144408 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.144290    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-tmp\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.144408 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.144328    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-metrics-client-ca\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.144584 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.144477    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.144584 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.144525    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.144584 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.144552    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.144584 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.144579    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvzzl\" (UniqueName: \"kubernetes.io/projected/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-kube-api-access-dvzzl\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.144864 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.144835    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-tmp\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.145059 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.144919    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-thanos-querier-tls\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.145385 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.145282    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-metrics-client-ca\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.147210 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.147184    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-grpc-tls\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.147553 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.147526    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.147701 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.147683    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.147811 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.147786    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.147865 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.147822    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-thanos-querier-tls\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.147954 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.147937    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.204059 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.204027    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvzzl\" (UniqueName: \"kubernetes.io/projected/65112dc9-5d05-4aa9-abcc-4eed6b6fb882-kube-api-access-dvzzl\") pod \"thanos-querier-6bb75f44d8-rclhz\" (UID: \"65112dc9-5d05-4aa9-abcc-4eed6b6fb882\") " pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.340670 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.340629    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:54.542029 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.541992    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75fdf7cdb5-fwp6s" event={"ID":"db9f14c4-8934-45ff-9330-4f57121b1464","Type":"ContainerStarted","Data":"4ac7b71e1e1035c04166f8c1f5375f3c81ad806689c8a1ae5fd308e14d530e1a"}
Oct 30 21:42:54.904374 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:54.904327    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"]
Oct 30 21:42:54.906340 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:54.906312    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65112dc9_5d05_4aa9_abcc_4eed6b6fb882.slice/crio-1477f7bad216f4950e7ae7397412f1b2ed4e02317aa499b8c4de4d076d0231e9 WatchSource:0}: Error finding container 1477f7bad216f4950e7ae7397412f1b2ed4e02317aa499b8c4de4d076d0231e9: Status 404 returned error can't find the container with id 1477f7bad216f4950e7ae7397412f1b2ed4e02317aa499b8c4de4d076d0231e9
Oct 30 21:42:55.552122 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:55.552079    4553 generic.go:356] "Generic (PLEG): container finished" podID="8ffe6dba-999b-4dec-ac96-20124d51b0f7" containerID="a148fd8a3dcc6e8c0b02d6bf2dd641dde696f6a8d7a728aa9229967e272673b0" exitCode=0
Oct 30 21:42:55.552325 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:55.552218    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbg2m" event={"ID":"8ffe6dba-999b-4dec-ac96-20124d51b0f7","Type":"ContainerDied","Data":"a148fd8a3dcc6e8c0b02d6bf2dd641dde696f6a8d7a728aa9229967e272673b0"}
Oct 30 21:42:55.555319 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:55.555290    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf" event={"ID":"62254772-614a-4bba-a726-6866cd578123","Type":"ContainerStarted","Data":"50139de7e4d4d9179d794b077b06dd97159e103690b9823d0c22370e028ce5ca"}
Oct 30 21:42:55.555446 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:55.555339    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf" event={"ID":"62254772-614a-4bba-a726-6866cd578123","Type":"ContainerStarted","Data":"ecd147c60205960e214ae488803d2cfc23f7a98deee902a244b83eead3775f65"}
Oct 30 21:42:55.555446 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:55.555359    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf" event={"ID":"62254772-614a-4bba-a726-6866cd578123","Type":"ContainerStarted","Data":"2e20d2b10cf58c74d8857044dc402d945fd292aa381a9da9277ca5a87a920c55"}
Oct 30 21:42:55.557360 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:55.557334    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz" event={"ID":"65112dc9-5d05-4aa9-abcc-4eed6b6fb882","Type":"ContainerStarted","Data":"1477f7bad216f4950e7ae7397412f1b2ed4e02317aa499b8c4de4d076d0231e9"}
Oct 30 21:42:55.561115 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:55.560600    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7" event={"ID":"c58cf9e5-68a2-4fd7-a364-d7a82686ee5d","Type":"ContainerStarted","Data":"033cc14d88f612ce82502cdc4bb767379565d24abaa288d60662eebdbe6899ff"}
Oct 30 21:42:55.790905 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:55.790852    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-rcbl7" podStartSLOduration=2.464111274 podStartE2EDuration="4.790837931s" totalImagesPullingTime="2.326726657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:55.790347193 +0000 UTC m=+26.524710228" watchObservedRunningTime="2026-10-30 21:42:55.790837931 +0000 UTC m=+26.525200956"
Oct 30 21:42:55.909102 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:55.909042    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-vjssf" podStartSLOduration=2.573344052 podStartE2EDuration="4.90902349s" totalImagesPullingTime="2.335679438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:55.908498099 +0000 UTC m=+26.642861134" watchObservedRunningTime="2026-10-30 21:42:55.90902349 +0000 UTC m=+26.643386522"
Oct 30 21:42:56.130872 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.130836    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-848d46785f-km9rb"]
Oct 30 21:42:56.133186 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.133162    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.164854 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.164814    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47537107-9197-4436-b13e-d675c5655e04-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.165033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.164881    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/47537107-9197-4436-b13e-d675c5655e04-secret-metrics-server-client-certs\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.165033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.164981    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/47537107-9197-4436-b13e-d675c5655e04-metrics-server-audit-profiles\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.165159 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.165032    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx6jm\" (UniqueName: \"kubernetes.io/projected/47537107-9197-4436-b13e-d675c5655e04-kube-api-access-qx6jm\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.165159 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.165070    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/47537107-9197-4436-b13e-d675c5655e04-client-ca-bundle\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.165159 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.165125    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/47537107-9197-4436-b13e-d675c5655e04-secret-metrics-server-tls\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.165265 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.165178    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/47537107-9197-4436-b13e-d675c5655e04-audit-log\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.186822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.186790    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 21:42:56.187017 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.186928    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vh4cg\""
Oct 30 21:42:56.187017 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.186964    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 21:42:56.187017 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.187011    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 21:42:56.187251 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.187173    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 21:42:56.187251 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.187188    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cfhddh8tie215\""
Oct 30 21:42:56.193553 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.193525    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-848d46785f-km9rb"]
Oct 30 21:42:56.270265 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.270209    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/47537107-9197-4436-b13e-d675c5655e04-secret-metrics-server-client-certs\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.270460 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.270294    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/47537107-9197-4436-b13e-d675c5655e04-metrics-server-audit-profiles\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.270460 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.270338    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx6jm\" (UniqueName: \"kubernetes.io/projected/47537107-9197-4436-b13e-d675c5655e04-kube-api-access-qx6jm\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.270460 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.270382    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/47537107-9197-4436-b13e-d675c5655e04-client-ca-bundle\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.270460 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.270445    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/47537107-9197-4436-b13e-d675c5655e04-secret-metrics-server-tls\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.270660 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.270498    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/47537107-9197-4436-b13e-d675c5655e04-audit-log\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.270660 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.270565    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47537107-9197-4436-b13e-d675c5655e04-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.271663 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.271634    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47537107-9197-4436-b13e-d675c5655e04-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.277593 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.273937    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/47537107-9197-4436-b13e-d675c5655e04-metrics-server-audit-profiles\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.277593 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.274635    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/47537107-9197-4436-b13e-d675c5655e04-audit-log\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.277593 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.274940    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/47537107-9197-4436-b13e-d675c5655e04-secret-metrics-server-client-certs\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.277593 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.275250    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/47537107-9197-4436-b13e-d675c5655e04-client-ca-bundle\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.279273 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.279248    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/47537107-9197-4436-b13e-d675c5655e04-secret-metrics-server-tls\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.336022 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.335989    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx6jm\" (UniqueName: \"kubernetes.io/projected/47537107-9197-4436-b13e-d675c5655e04-kube-api-access-qx6jm\") pod \"metrics-server-848d46785f-km9rb\" (UID: \"47537107-9197-4436-b13e-d675c5655e04\") " pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.446223 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.446124    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:42:56.568024 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.567986    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbg2m" event={"ID":"8ffe6dba-999b-4dec-ac96-20124d51b0f7","Type":"ContainerStarted","Data":"e80f36555040691aa6881918928f459e0e718ac3054b1ee6c5796b8f845de5e8"}
Oct 30 21:42:56.568223 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.568078    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbg2m" event={"ID":"8ffe6dba-999b-4dec-ac96-20124d51b0f7","Type":"ContainerStarted","Data":"cc6761dd0acd4ef6d8271cd3dc28ecb3c4bd3c5c9f6839bd50668c079b827add"}
Oct 30 21:42:56.686617 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.686208    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sbg2m" podStartSLOduration=2.884172876 podStartE2EDuration="5.686193745s" totalImagesPullingTime="2.698415404s" totalInitContainerRuntime="103.605465ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:56.685316265 +0000 UTC m=+27.419679302" watchObservedRunningTime="2026-10-30 21:42:56.686193745 +0000 UTC m=+27.420556777"
Oct 30 21:42:56.874989 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:56.874908    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-848d46785f-km9rb"]
Oct 30 21:42:56.878766 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:42:56.878724    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47537107_9197_4436_b13e_d675c5655e04.slice/crio-0d2f0e76fc048848786d10af8ebf201c132b9afab556bbd3290b036eb457c813 WatchSource:0}: Error finding container 0d2f0e76fc048848786d10af8ebf201c132b9afab556bbd3290b036eb457c813: Status 404 returned error can't find the container with id 0d2f0e76fc048848786d10af8ebf201c132b9afab556bbd3290b036eb457c813
Oct 30 21:42:57.573245 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:57.573204    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75fdf7cdb5-fwp6s" event={"ID":"db9f14c4-8934-45ff-9330-4f57121b1464","Type":"ContainerStarted","Data":"69e1cea5c9a9a08260761f847536b75a7ed47033405bd7a2f3e9e11d68c5f814"}
Oct 30 21:42:57.575725 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:57.575701    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz" event={"ID":"65112dc9-5d05-4aa9-abcc-4eed6b6fb882","Type":"ContainerStarted","Data":"51065f3297c08fe9af89e4c2aa555f010f36bb0f074a145cc6b999a3a7e03bb9"}
Oct 30 21:42:57.575849 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:57.575756    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz" event={"ID":"65112dc9-5d05-4aa9-abcc-4eed6b6fb882","Type":"ContainerStarted","Data":"9beb85f98f243ad0fd68b02065a84c9fc3571eb422652ceab9ea4cb5ea0d789d"}
Oct 30 21:42:57.575849 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:57.575790    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz" event={"ID":"65112dc9-5d05-4aa9-abcc-4eed6b6fb882","Type":"ContainerStarted","Data":"5e4c79191b0cff5e040829fe1fad4dd48e7266cf044dfd833661f533182cead2"}
Oct 30 21:42:57.576844 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:57.576822    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-848d46785f-km9rb" event={"ID":"47537107-9197-4436-b13e-d675c5655e04","Type":"ContainerStarted","Data":"0d2f0e76fc048848786d10af8ebf201c132b9afab556bbd3290b036eb457c813"}
Oct 30 21:42:57.688906 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:57.688844    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-75fdf7cdb5-fwp6s" podStartSLOduration=2.211987813 podStartE2EDuration="5.688827537s" totalImagesPullingTime="3.476839724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:57.688301051 +0000 UTC m=+28.422664099" watchObservedRunningTime="2026-10-30 21:42:57.688827537 +0000 UTC m=+28.423190569"
Oct 30 21:42:58.203758 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:42:58.203721    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:42:59.589347 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:59.589308    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz" event={"ID":"65112dc9-5d05-4aa9-abcc-4eed6b6fb882","Type":"ContainerStarted","Data":"d033a6bddb1c5c801240e0b018e4b41c770da3a10ca3104e0afb0e65fbd490cf"}
Oct 30 21:42:59.589816 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:59.589393    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz" event={"ID":"65112dc9-5d05-4aa9-abcc-4eed6b6fb882","Type":"ContainerStarted","Data":"1a6c78fe4fe3662ef2187be4514e095b561adf5232a63415cc37f155dffcc488"}
Oct 30 21:42:59.589816 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:59.589442    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz" event={"ID":"65112dc9-5d05-4aa9-abcc-4eed6b6fb882","Type":"ContainerStarted","Data":"24d9f6f51b7b15ae2a55e26d57181da5dbf247b0752bba2d09985e5de21870a5"}
Oct 30 21:42:59.589816 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:59.589524    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:42:59.591010 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:59.590987    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-848d46785f-km9rb" event={"ID":"47537107-9197-4436-b13e-d675c5655e04","Type":"ContainerStarted","Data":"6c0b509110fbb8170fa4569315f1da2a02354391ea2dc15f835f211b6aaaf931"}
Oct 30 21:42:59.660028 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:59.659998    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7d5lt"
Oct 30 21:42:59.811861 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:59.811802    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz" podStartSLOduration=3.171048425 podStartE2EDuration="6.811789051s" totalImagesPullingTime="3.640740626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:59.761254111 +0000 UTC m=+30.495617144" watchObservedRunningTime="2026-10-30 21:42:59.811789051 +0000 UTC m=+30.546152081"
Oct 30 21:42:59.996594 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:42:59.996495    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-848d46785f-km9rb" podStartSLOduration=2.24638368 podStartE2EDuration="3.996480831s" totalImagesPullingTime="1.750097151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:59.995536123 +0000 UTC m=+30.729899157" watchObservedRunningTime="2026-10-30 21:42:59.996480831 +0000 UTC m=+30.730843862"
Oct 30 21:43:03.333782 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.333737    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:43:03.334202 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.333803    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:43:03.338591 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.338565    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:43:03.619293 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.619213    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:43:03.652051 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.652018    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75fdf7cdb5-fwp6s"]
Oct 30 21:43:03.687596 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.687564    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-655b7777d9-k7pr7"]
Oct 30 21:43:03.697683 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.697659    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.744097 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.744067    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-oauth-serving-cert\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.744097 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.744096    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzbsw\" (UniqueName: \"kubernetes.io/projected/803056fa-dcc5-43be-8136-a0554d7e3b7a-kube-api-access-nzbsw\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.744370 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.744151    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-serving-cert\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.744370 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.744282    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/803056fa-dcc5-43be-8136-a0554d7e3b7a-tmp\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.744370 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.744332    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-service-ca\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.744521 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.744395    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-config\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.744521 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.744474    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-trusted-ca-bundle\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.744792 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.744767    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-oauth-config\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.749326 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.749303    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-655b7777d9-k7pr7"]
Oct 30 21:43:03.846217 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.846184    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-oauth-config\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.846404 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.846227    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-oauth-serving-cert\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.846404 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.846243    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzbsw\" (UniqueName: \"kubernetes.io/projected/803056fa-dcc5-43be-8136-a0554d7e3b7a-kube-api-access-nzbsw\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.846404 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.846271    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-serving-cert\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.846404 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.846299    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/803056fa-dcc5-43be-8136-a0554d7e3b7a-tmp\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.846404 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.846322    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-service-ca\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.846404 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.846356    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-config\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.846404 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.846387    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-trusted-ca-bundle\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.847221 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.847192    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-config\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.847288 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.847224    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-oauth-serving-cert\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.847369 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.847347    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-service-ca\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.847413 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.847376    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/803056fa-dcc5-43be-8136-a0554d7e3b7a-tmp\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.847864 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.847846    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-trusted-ca-bundle\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.849279 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.849258    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-serving-cert\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.849377 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.849360    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-oauth-config\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:03.904628 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:03.904561    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzbsw\" (UniqueName: \"kubernetes.io/projected/803056fa-dcc5-43be-8136-a0554d7e3b7a-kube-api-access-nzbsw\") pod \"console-655b7777d9-k7pr7\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") " pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:04.020946 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:04.020901    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:04.428326 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:04.428302    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-655b7777d9-k7pr7"]
Oct 30 21:43:04.430649 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:43:04.430618    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803056fa_dcc5_43be_8136_a0554d7e3b7a.slice/crio-533f606ae9cc04459e7c013295806eae9b31b1b6965274ffce8d33f10cbbe057 WatchSource:0}: Error finding container 533f606ae9cc04459e7c013295806eae9b31b1b6965274ffce8d33f10cbbe057: Status 404 returned error can't find the container with id 533f606ae9cc04459e7c013295806eae9b31b1b6965274ffce8d33f10cbbe057
Oct 30 21:43:04.625725 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:04.625683    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655b7777d9-k7pr7" event={"ID":"803056fa-dcc5-43be-8136-a0554d7e3b7a","Type":"ContainerStarted","Data":"1dfc1e106f27756980de5c5c2f5c538b47fce88aac12fa5dcd20c2e3f8152fea"}
Oct 30 21:43:04.625917 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:04.625780    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655b7777d9-k7pr7" event={"ID":"803056fa-dcc5-43be-8136-a0554d7e3b7a","Type":"ContainerStarted","Data":"533f606ae9cc04459e7c013295806eae9b31b1b6965274ffce8d33f10cbbe057"}
Oct 30 21:43:04.750657 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:04.750555    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-655b7777d9-k7pr7" podStartSLOduration=1.7505388640000001 podStartE2EDuration="1.750538864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:04.750037589 +0000 UTC m=+35.484400620" watchObservedRunningTime="2026-10-30 21:43:04.750538864 +0000 UTC m=+35.484901895"
Oct 30 21:43:05.611904 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:05.611877    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6bb75f44d8-rclhz"
Oct 30 21:43:14.021971 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:14.021880    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:14.022428 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:14.022175    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:14.026649 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:14.026629    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:14.666677 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:14.666643    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:43:16.446306 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:16.446271    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:43:16.446702 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:16.446331    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:43:28.200339 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:43:28.200299    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:43:30.649335 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.649271    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-75fdf7cdb5-fwp6s" podUID="db9f14c4-8934-45ff-9330-4f57121b1464" containerName="console" containerID="cri-o://69e1cea5c9a9a08260761f847536b75a7ed47033405bd7a2f3e9e11d68c5f814" gracePeriod=15
Oct 30 21:43:30.900074 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.900014    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:43:30.979752 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.979718    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-service-ca\") pod \"db9f14c4-8934-45ff-9330-4f57121b1464\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") "
Oct 30 21:43:30.979919 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.979794    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db9f14c4-8934-45ff-9330-4f57121b1464-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db9f14c4-8934-45ff-9330-4f57121b1464-console-serving-cert\") pod \"db9f14c4-8934-45ff-9330-4f57121b1464\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") "
Oct 30 21:43:30.979919 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.979841    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-trusted-ca-bundle\") pod \"db9f14c4-8934-45ff-9330-4f57121b1464\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") "
Oct 30 21:43:30.979919 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.979878    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db9f14c4-8934-45ff-9330-4f57121b1464-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db9f14c4-8934-45ff-9330-4f57121b1464-console-oauth-config\") pod \"db9f14c4-8934-45ff-9330-4f57121b1464\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") "
Oct 30 21:43:30.979919 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.979908    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-oauth-serving-cert\") pod \"db9f14c4-8934-45ff-9330-4f57121b1464\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") "
Oct 30 21:43:30.980095 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.979937    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9f14c4-8934-45ff-9330-4f57121b1464-tmp\" (UniqueName: \"kubernetes.io/empty-dir/db9f14c4-8934-45ff-9330-4f57121b1464-tmp\") pod \"db9f14c4-8934-45ff-9330-4f57121b1464\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") "
Oct 30 21:43:30.980174 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.980090    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-console-config\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-console-config\") pod \"db9f14c4-8934-45ff-9330-4f57121b1464\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") "
Oct 30 21:43:30.980239 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.980175    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db9f14c4-8934-45ff-9330-4f57121b1464-kube-api-access-z8jpk\" (UniqueName: \"kubernetes.io/projected/db9f14c4-8934-45ff-9330-4f57121b1464-kube-api-access-z8jpk\") pod \"db9f14c4-8934-45ff-9330-4f57121b1464\" (UID: \"db9f14c4-8934-45ff-9330-4f57121b1464\") "
Oct 30 21:43:30.980295 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.980267    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9f14c4-8934-45ff-9330-4f57121b1464-tmp" pod "db9f14c4-8934-45ff-9330-4f57121b1464" (UID: "db9f14c4-8934-45ff-9330-4f57121b1464"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:30.980354 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.980305    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-service-ca" pod "db9f14c4-8934-45ff-9330-4f57121b1464" (UID: "db9f14c4-8934-45ff-9330-4f57121b1464"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:30.980482 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.980454    4553 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db9f14c4-8934-45ff-9330-4f57121b1464-tmp\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:30.980482 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.980477    4553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:30.980664 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.980436    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-trusted-ca-bundle" pod "db9f14c4-8934-45ff-9330-4f57121b1464" (UID: "db9f14c4-8934-45ff-9330-4f57121b1464"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:30.980664 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.980609    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-console-config" pod "db9f14c4-8934-45ff-9330-4f57121b1464" (UID: "db9f14c4-8934-45ff-9330-4f57121b1464"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:30.980817 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.980799    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-oauth-serving-cert" pod "db9f14c4-8934-45ff-9330-4f57121b1464" (UID: "db9f14c4-8934-45ff-9330-4f57121b1464"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:30.982629 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.982611    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9f14c4-8934-45ff-9330-4f57121b1464-kube-api-access-z8jpk" pod "db9f14c4-8934-45ff-9330-4f57121b1464" (UID: "db9f14c4-8934-45ff-9330-4f57121b1464"). InnerVolumeSpecName "kube-api-access-z8jpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:30.982715 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.982701    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db9f14c4-8934-45ff-9330-4f57121b1464-console-oauth-config" pod "db9f14c4-8934-45ff-9330-4f57121b1464" (UID: "db9f14c4-8934-45ff-9330-4f57121b1464"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:30.982759 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:30.982723    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db9f14c4-8934-45ff-9330-4f57121b1464-console-serving-cert" pod "db9f14c4-8934-45ff-9330-4f57121b1464" (UID: "db9f14c4-8934-45ff-9330-4f57121b1464"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:31.081384 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:31.081352    4553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-trusted-ca-bundle\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:31.081384 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:31.081378    4553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db9f14c4-8934-45ff-9330-4f57121b1464-console-oauth-config\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:31.081384 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:31.081389    4553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-oauth-serving-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:31.081622 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:31.081399    4553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db9f14c4-8934-45ff-9330-4f57121b1464-console-config\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:31.081622 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:31.081409    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8jpk\" (UniqueName: \"kubernetes.io/projected/db9f14c4-8934-45ff-9330-4f57121b1464-kube-api-access-z8jpk\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:31.081622 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:31.081419    4553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db9f14c4-8934-45ff-9330-4f57121b1464-console-serving-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:31.724337 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:31.724254    4553 generic.go:356] "Generic (PLEG): container finished" podID="db9f14c4-8934-45ff-9330-4f57121b1464" containerID="69e1cea5c9a9a08260761f847536b75a7ed47033405bd7a2f3e9e11d68c5f814" exitCode=0
Oct 30 21:43:31.724337 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:31.724303    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75fdf7cdb5-fwp6s"
Oct 30 21:43:31.724337 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:31.724321    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75fdf7cdb5-fwp6s" event={"ID":"db9f14c4-8934-45ff-9330-4f57121b1464","Type":"ContainerDied","Data":"69e1cea5c9a9a08260761f847536b75a7ed47033405bd7a2f3e9e11d68c5f814"}
Oct 30 21:43:31.724806 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:31.724363    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75fdf7cdb5-fwp6s" event={"ID":"db9f14c4-8934-45ff-9330-4f57121b1464","Type":"ContainerDied","Data":"4ac7b71e1e1035c04166f8c1f5375f3c81ad806689c8a1ae5fd308e14d530e1a"}
Oct 30 21:43:31.724806 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:31.724382    4553 scope.go:122] "RemoveContainer" containerID="69e1cea5c9a9a08260761f847536b75a7ed47033405bd7a2f3e9e11d68c5f814"
Oct 30 21:43:31.732546 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:31.732525    4553 scope.go:122] "RemoveContainer" containerID="69e1cea5c9a9a08260761f847536b75a7ed47033405bd7a2f3e9e11d68c5f814"
Oct 30 21:43:31.732815 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:43:31.732794    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e1cea5c9a9a08260761f847536b75a7ed47033405bd7a2f3e9e11d68c5f814\": container with ID starting with 69e1cea5c9a9a08260761f847536b75a7ed47033405bd7a2f3e9e11d68c5f814 not found: ID does not exist" containerID="69e1cea5c9a9a08260761f847536b75a7ed47033405bd7a2f3e9e11d68c5f814"
Oct 30 21:43:31.732872 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:31.732826    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e1cea5c9a9a08260761f847536b75a7ed47033405bd7a2f3e9e11d68c5f814"} err="failed to get container status \"69e1cea5c9a9a08260761f847536b75a7ed47033405bd7a2f3e9e11d68c5f814\": rpc error: code = NotFound desc = could not find container \"69e1cea5c9a9a08260761f847536b75a7ed47033405bd7a2f3e9e11d68c5f814\": container with ID starting with 69e1cea5c9a9a08260761f847536b75a7ed47033405bd7a2f3e9e11d68c5f814 not found: ID does not exist"
Oct 30 21:43:31.891771 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:31.891734    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75fdf7cdb5-fwp6s"]
Oct 30 21:43:31.942159 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:31.942109    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75fdf7cdb5-fwp6s"]
Oct 30 21:43:33.391455 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:33.391426    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db9f14c4-8934-45ff-9330-4f57121b1464" path="/var/lib/kubelet/pods/db9f14c4-8934-45ff-9330-4f57121b1464/volumes"
Oct 30 21:43:36.451477 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:36.451453    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:43:36.455278 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:36.455256    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-848d46785f-km9rb"
Oct 30 21:43:47.781908 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:47.781878    4553 generic.go:356] "Generic (PLEG): container finished" podID="47734525-f46d-41de-93b8-c8b8f4087651" containerID="371a295b33c21af65143447f555f780b7ddee037bfda841824c6efbddb79883f" exitCode=0
Oct 30 21:43:47.782330 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:47.781954    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fpxlc-s8s8f" event={"ID":"47734525-f46d-41de-93b8-c8b8f4087651","Type":"ContainerDied","Data":"371a295b33c21af65143447f555f780b7ddee037bfda841824c6efbddb79883f"}
Oct 30 21:43:48.933581 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:48.933550    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:43:49.018262 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.018225    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47734525-f46d-41de-93b8-c8b8f4087651-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/47734525-f46d-41de-93b8-c8b8f4087651-archives-path\") pod \"47734525-f46d-41de-93b8-c8b8f4087651\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") "
Oct 30 21:43:49.018454 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.018272    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-trusted-ca-bundle\") pod \"47734525-f46d-41de-93b8-c8b8f4087651\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") "
Oct 30 21:43:49.018454 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.018311    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-service-ca-bundle\") pod \"47734525-f46d-41de-93b8-c8b8f4087651\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") "
Oct 30 21:43:49.018454 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.018381    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47734525-f46d-41de-93b8-c8b8f4087651-kube-api-access-2d6fv\" (UniqueName: \"kubernetes.io/projected/47734525-f46d-41de-93b8-c8b8f4087651-kube-api-access-2d6fv\") pod \"47734525-f46d-41de-93b8-c8b8f4087651\" (UID: \"47734525-f46d-41de-93b8-c8b8f4087651\") "
Oct 30 21:43:49.018692 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.018671    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47734525-f46d-41de-93b8-c8b8f4087651-archives-path" pod "47734525-f46d-41de-93b8-c8b8f4087651" (UID: "47734525-f46d-41de-93b8-c8b8f4087651"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:43:49.018764 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.018729    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-service-ca-bundle" pod "47734525-f46d-41de-93b8-c8b8f4087651" (UID: "47734525-f46d-41de-93b8-c8b8f4087651"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:49.018764 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.018740    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-trusted-ca-bundle" pod "47734525-f46d-41de-93b8-c8b8f4087651" (UID: "47734525-f46d-41de-93b8-c8b8f4087651"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:49.020857 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.020839    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47734525-f46d-41de-93b8-c8b8f4087651-kube-api-access-2d6fv" pod "47734525-f46d-41de-93b8-c8b8f4087651" (UID: "47734525-f46d-41de-93b8-c8b8f4087651"). InnerVolumeSpecName "kube-api-access-2d6fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:49.119389 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.119360    4553 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/47734525-f46d-41de-93b8-c8b8f4087651-archives-path\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:49.119389 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.119387    4553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-trusted-ca-bundle\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:49.119581 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.119399    4553 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/47734525-f46d-41de-93b8-c8b8f4087651-service-ca-bundle\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:49.119581 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.119408    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d6fv\" (UniqueName: \"kubernetes.io/projected/47734525-f46d-41de-93b8-c8b8f4087651-kube-api-access-2d6fv\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:49.233848 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.233817    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-fpxlc-s8s8f"]
Oct 30 21:43:49.238099 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.238075    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-fpxlc-s8s8f"]
Oct 30 21:43:49.390914 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.390838    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47734525-f46d-41de-93b8-c8b8f4087651" path="/var/lib/kubelet/pods/47734525-f46d-41de-93b8-c8b8f4087651/volumes"
Oct 30 21:43:49.790455 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.790376    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fpxlc-s8s8f"
Oct 30 21:43:49.790611 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:43:49.790383    4553 scope.go:122] "RemoveContainer" containerID="371a295b33c21af65143447f555f780b7ddee037bfda841824c6efbddb79883f"
Oct 30 21:43:58.198949 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:43:58.198909    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:44:06.777361 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.777326    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-668777f575-xqcvb"]
Oct 30 21:44:06.777985 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.777648    4553 cpu_manager.go:414] "containerMap: removing container" podUID="47734525-f46d-41de-93b8-c8b8f4087651" containerName="insights-gathering"
Oct 30 21:44:06.777985 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.777661    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47734525-f46d-41de-93b8-c8b8f4087651" containerName="insights-gathering"
Oct 30 21:44:06.777985 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.777683    4553 cpu_manager.go:414] "containerMap: removing container" podUID="db9f14c4-8934-45ff-9330-4f57121b1464" containerName="console"
Oct 30 21:44:06.777985 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.777688    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db9f14c4-8934-45ff-9330-4f57121b1464" containerName="console"
Oct 30 21:44:06.777985 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.777742    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="db9f14c4-8934-45ff-9330-4f57121b1464" containerName="console"
Oct 30 21:44:06.777985 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.777754    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="47734525-f46d-41de-93b8-c8b8f4087651" containerName="insights-gathering"
Oct 30 21:44:06.780571 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.780551    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.857457 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.857417    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd9cf18-0334-4788-941b-78a7a6b22b20-service-ca\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.857620 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.857466    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bd9cf18-0334-4788-941b-78a7a6b22b20-tmp\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.857620 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.857536    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz8wp\" (UniqueName: \"kubernetes.io/projected/5bd9cf18-0334-4788-941b-78a7a6b22b20-kube-api-access-pz8wp\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.857620 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.857591    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bd9cf18-0334-4788-941b-78a7a6b22b20-console-serving-cert\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.857725 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.857620    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bd9cf18-0334-4788-941b-78a7a6b22b20-console-config\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.857725 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.857650    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bd9cf18-0334-4788-941b-78a7a6b22b20-console-oauth-config\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.857790 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.857743    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bd9cf18-0334-4788-941b-78a7a6b22b20-trusted-ca-bundle\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.857790 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.857760    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bd9cf18-0334-4788-941b-78a7a6b22b20-oauth-serving-cert\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.886241 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.886204    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-668777f575-xqcvb"]
Oct 30 21:44:06.958126 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.958088    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bd9cf18-0334-4788-941b-78a7a6b22b20-trusted-ca-bundle\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.958126 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.958147    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bd9cf18-0334-4788-941b-78a7a6b22b20-oauth-serving-cert\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.958418 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.958176    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd9cf18-0334-4788-941b-78a7a6b22b20-service-ca\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.958418 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.958197    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bd9cf18-0334-4788-941b-78a7a6b22b20-tmp\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.958418 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.958222    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz8wp\" (UniqueName: \"kubernetes.io/projected/5bd9cf18-0334-4788-941b-78a7a6b22b20-kube-api-access-pz8wp\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.958418 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.958244    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bd9cf18-0334-4788-941b-78a7a6b22b20-console-serving-cert\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.958418 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.958276    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bd9cf18-0334-4788-941b-78a7a6b22b20-console-config\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.958418 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.958312    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bd9cf18-0334-4788-941b-78a7a6b22b20-console-oauth-config\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.958710 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.958688    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bd9cf18-0334-4788-941b-78a7a6b22b20-tmp\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.959056 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.959029    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd9cf18-0334-4788-941b-78a7a6b22b20-service-ca\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.959174 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.959128    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5bd9cf18-0334-4788-941b-78a7a6b22b20-oauth-serving-cert\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.959240 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.959172    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5bd9cf18-0334-4788-941b-78a7a6b22b20-console-config\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.959240 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.959157    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bd9cf18-0334-4788-941b-78a7a6b22b20-trusted-ca-bundle\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.960973 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.960955    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5bd9cf18-0334-4788-941b-78a7a6b22b20-console-serving-cert\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:06.961020 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:06.961008    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5bd9cf18-0334-4788-941b-78a7a6b22b20-console-oauth-config\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:07.017011 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:07.016984    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz8wp\" (UniqueName: \"kubernetes.io/projected/5bd9cf18-0334-4788-941b-78a7a6b22b20-kube-api-access-pz8wp\") pod \"console-668777f575-xqcvb\" (UID: \"5bd9cf18-0334-4788-941b-78a7a6b22b20\") " pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:07.090627 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:07.090589    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:07.460836 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:07.460755    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-668777f575-xqcvb"]
Oct 30 21:44:07.464146 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:44:07.464101    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd9cf18_0334_4788_941b_78a7a6b22b20.slice/crio-c2f664f900bd0046548ab7c662262de3fd7b60f11936d40517c7a2381430f954 WatchSource:0}: Error finding container c2f664f900bd0046548ab7c662262de3fd7b60f11936d40517c7a2381430f954: Status 404 returned error can't find the container with id c2f664f900bd0046548ab7c662262de3fd7b60f11936d40517c7a2381430f954
Oct 30 21:44:07.854916 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:07.854873    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668777f575-xqcvb" event={"ID":"5bd9cf18-0334-4788-941b-78a7a6b22b20","Type":"ContainerStarted","Data":"ab3eb4feebe6769a6910a017bdc9bbff09cbb4e550baca020228b11179b000c7"}
Oct 30 21:44:07.855427 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:07.854930    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668777f575-xqcvb" event={"ID":"5bd9cf18-0334-4788-941b-78a7a6b22b20","Type":"ContainerStarted","Data":"c2f664f900bd0046548ab7c662262de3fd7b60f11936d40517c7a2381430f954"}
Oct 30 21:44:08.030308 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:08.030223    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-668777f575-xqcvb" podStartSLOduration=2.030203717 podStartE2EDuration="2.030203717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:07.979588139 +0000 UTC m=+98.713951170" watchObservedRunningTime="2026-10-30 21:44:08.030203717 +0000 UTC m=+98.764566742"
Oct 30 21:44:17.091665 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:17.091623    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:17.092053 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:17.091726    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:17.096383 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:17.096360    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:17.899454 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:17.899427    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-668777f575-xqcvb"
Oct 30 21:44:18.071958 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:18.071927    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-655b7777d9-k7pr7"]
Oct 30 21:44:28.199770 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:44:28.199731    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:44:43.092355 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.092314    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-655b7777d9-k7pr7" podUID="803056fa-dcc5-43be-8136-a0554d7e3b7a" containerName="console" containerID="cri-o://1dfc1e106f27756980de5c5c2f5c538b47fce88aac12fa5dcd20c2e3f8152fea" gracePeriod=15
Oct 30 21:44:43.336409 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.336382    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:44:43.364810 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.364733    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-serving-cert\") pod \"803056fa-dcc5-43be-8136-a0554d7e3b7a\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") "
Oct 30 21:44:43.364810 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.364775    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803056fa-dcc5-43be-8136-a0554d7e3b7a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/803056fa-dcc5-43be-8136-a0554d7e3b7a-tmp\") pod \"803056fa-dcc5-43be-8136-a0554d7e3b7a\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") "
Oct 30 21:44:43.364810 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.364799    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803056fa-dcc5-43be-8136-a0554d7e3b7a-kube-api-access-nzbsw\" (UniqueName: \"kubernetes.io/projected/803056fa-dcc5-43be-8136-a0554d7e3b7a-kube-api-access-nzbsw\") pod \"803056fa-dcc5-43be-8136-a0554d7e3b7a\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") "
Oct 30 21:44:43.365057 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.364826    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-trusted-ca-bundle\") pod \"803056fa-dcc5-43be-8136-a0554d7e3b7a\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") "
Oct 30 21:44:43.365057 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.364857    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-oauth-config\") pod \"803056fa-dcc5-43be-8136-a0554d7e3b7a\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") "
Oct 30 21:44:43.365057 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.364888    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-oauth-serving-cert\") pod \"803056fa-dcc5-43be-8136-a0554d7e3b7a\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") "
Oct 30 21:44:43.365057 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.364926    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-config\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-config\") pod \"803056fa-dcc5-43be-8136-a0554d7e3b7a\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") "
Oct 30 21:44:43.365275 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.365045    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803056fa-dcc5-43be-8136-a0554d7e3b7a-tmp" pod "803056fa-dcc5-43be-8136-a0554d7e3b7a" (UID: "803056fa-dcc5-43be-8136-a0554d7e3b7a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:43.365275 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.365066    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-service-ca\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-service-ca\") pod \"803056fa-dcc5-43be-8136-a0554d7e3b7a\" (UID: \"803056fa-dcc5-43be-8136-a0554d7e3b7a\") "
Oct 30 21:44:43.365381 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.365357    4553 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/803056fa-dcc5-43be-8136-a0554d7e3b7a-tmp\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:43.365436 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.365405    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-config" pod "803056fa-dcc5-43be-8136-a0554d7e3b7a" (UID: "803056fa-dcc5-43be-8136-a0554d7e3b7a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:43.365436 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.365419    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-trusted-ca-bundle" pod "803056fa-dcc5-43be-8136-a0554d7e3b7a" (UID: "803056fa-dcc5-43be-8136-a0554d7e3b7a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:43.365546 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.365436    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-oauth-serving-cert" pod "803056fa-dcc5-43be-8136-a0554d7e3b7a" (UID: "803056fa-dcc5-43be-8136-a0554d7e3b7a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:43.365755 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.365734    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-service-ca" pod "803056fa-dcc5-43be-8136-a0554d7e3b7a" (UID: "803056fa-dcc5-43be-8136-a0554d7e3b7a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:44:43.367189 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.367167    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803056fa-dcc5-43be-8136-a0554d7e3b7a-kube-api-access-nzbsw" pod "803056fa-dcc5-43be-8136-a0554d7e3b7a" (UID: "803056fa-dcc5-43be-8136-a0554d7e3b7a"). InnerVolumeSpecName "kube-api-access-nzbsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:43.367267 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.367215    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-serving-cert" pod "803056fa-dcc5-43be-8136-a0554d7e3b7a" (UID: "803056fa-dcc5-43be-8136-a0554d7e3b7a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:43.367348 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.367325    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-oauth-config" pod "803056fa-dcc5-43be-8136-a0554d7e3b7a" (UID: "803056fa-dcc5-43be-8136-a0554d7e3b7a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:44:43.466365 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.466326    4553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-serving-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:43.466365 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.466354    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzbsw\" (UniqueName: \"kubernetes.io/projected/803056fa-dcc5-43be-8136-a0554d7e3b7a-kube-api-access-nzbsw\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:43.466365 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.466364    4553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-trusted-ca-bundle\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:43.466583 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.466375    4553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-oauth-config\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:43.466583 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.466389    4553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-oauth-serving-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:43.466583 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.466398    4553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-console-config\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:43.466583 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.466407    4553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/803056fa-dcc5-43be-8136-a0554d7e3b7a-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:44:43.986612 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.986578    4553 generic.go:356] "Generic (PLEG): container finished" podID="803056fa-dcc5-43be-8136-a0554d7e3b7a" containerID="1dfc1e106f27756980de5c5c2f5c538b47fce88aac12fa5dcd20c2e3f8152fea" exitCode=0
Oct 30 21:44:43.986612 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.986618    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-655b7777d9-k7pr7"
Oct 30 21:44:43.986834 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.986666    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655b7777d9-k7pr7" event={"ID":"803056fa-dcc5-43be-8136-a0554d7e3b7a","Type":"ContainerDied","Data":"1dfc1e106f27756980de5c5c2f5c538b47fce88aac12fa5dcd20c2e3f8152fea"}
Oct 30 21:44:43.986834 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.986720    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-655b7777d9-k7pr7" event={"ID":"803056fa-dcc5-43be-8136-a0554d7e3b7a","Type":"ContainerDied","Data":"533f606ae9cc04459e7c013295806eae9b31b1b6965274ffce8d33f10cbbe057"}
Oct 30 21:44:43.986834 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.986744    4553 scope.go:122] "RemoveContainer" containerID="1dfc1e106f27756980de5c5c2f5c538b47fce88aac12fa5dcd20c2e3f8152fea"
Oct 30 21:44:43.994735 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.994714    4553 scope.go:122] "RemoveContainer" containerID="1dfc1e106f27756980de5c5c2f5c538b47fce88aac12fa5dcd20c2e3f8152fea"
Oct 30 21:44:43.995011 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:44:43.994994    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfc1e106f27756980de5c5c2f5c538b47fce88aac12fa5dcd20c2e3f8152fea\": container with ID starting with 1dfc1e106f27756980de5c5c2f5c538b47fce88aac12fa5dcd20c2e3f8152fea not found: ID does not exist" containerID="1dfc1e106f27756980de5c5c2f5c538b47fce88aac12fa5dcd20c2e3f8152fea"
Oct 30 21:44:43.995078 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:43.995021    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfc1e106f27756980de5c5c2f5c538b47fce88aac12fa5dcd20c2e3f8152fea"} err="failed to get container status \"1dfc1e106f27756980de5c5c2f5c538b47fce88aac12fa5dcd20c2e3f8152fea\": rpc error: code = NotFound desc = could not find container \"1dfc1e106f27756980de5c5c2f5c538b47fce88aac12fa5dcd20c2e3f8152fea\": container with ID starting with 1dfc1e106f27756980de5c5c2f5c538b47fce88aac12fa5dcd20c2e3f8152fea not found: ID does not exist"
Oct 30 21:44:44.153016 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:44.152978    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-655b7777d9-k7pr7"]
Oct 30 21:44:44.203668 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:44.203635    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-655b7777d9-k7pr7"]
Oct 30 21:44:45.389852 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:44:45.389820    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803056fa-dcc5-43be-8136-a0554d7e3b7a" path="/var/lib/kubelet/pods/803056fa-dcc5-43be-8136-a0554d7e3b7a/volumes"
Oct 30 21:44:58.199425 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:44:58.199386    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:45:28.200255 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:45:28.200216    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:45:58.200959 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:45:58.200924    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:46:00.144767 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.144729    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"]
Oct 30 21:46:00.145290 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.145184    4553 cpu_manager.go:414] "containerMap: removing container" podUID="803056fa-dcc5-43be-8136-a0554d7e3b7a" containerName="console"
Oct 30 21:46:00.145290 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.145203    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803056fa-dcc5-43be-8136-a0554d7e3b7a" containerName="console"
Oct 30 21:46:00.145404 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.145302    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="803056fa-dcc5-43be-8136-a0554d7e3b7a" containerName="console"
Oct 30 21:46:00.148216 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.148193    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.208363 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.208328    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 21:46:00.211835 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.211812    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 21:46:00.212040 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.212002    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4nvfx\""
Oct 30 21:46:00.255967 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.255937    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 21:46:00.256203 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.256028    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:46:00.256203 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.256032    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:46:00.256203 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.256090    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:46:00.257113 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.257091    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"]
Oct 30 21:46:00.265639 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.265610    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.265736 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.265656    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.265736 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.265675    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpmc7\" (UniqueName: \"kubernetes.io/projected/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-kube-api-access-vpmc7\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.265831 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.265814    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.265866 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.265856    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.265920 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.265903    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.265954 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.265942    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.366654 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.366614    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.366654 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.366655    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.366920 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.366680    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.366920 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.366808    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.366920 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.366865    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.366920 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.366887    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpmc7\" (UniqueName: \"kubernetes.io/projected/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-kube-api-access-vpmc7\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.367126 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.366936    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.367385 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.367354    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.369417 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.369394    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.369529 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.369396    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.369779 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.369754    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.369859 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.369803    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.424932 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.424863    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.424932 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.424870    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpmc7\" (UniqueName: \"kubernetes.io/projected/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-kube-api-access-vpmc7\") pod \"istiod-openshift-gateway-7876bfcb74-xpm8h\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.459959 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.459923    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:00.883212 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:00.883181    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"]
Oct 30 21:46:00.886481 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:46:00.886449    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c61bb46_57bd_4a99_9ea6_ac5a98302d73.slice/crio-434d4611ef8e1a0eec43e0ffcaa831bc9899c8d948ed66a1feb60c7073aaec23 WatchSource:0}: Error finding container 434d4611ef8e1a0eec43e0ffcaa831bc9899c8d948ed66a1feb60c7073aaec23: Status 404 returned error can't find the container with id 434d4611ef8e1a0eec43e0ffcaa831bc9899c8d948ed66a1feb60c7073aaec23
Oct 30 21:46:01.261535 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:01.261454    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h" event={"ID":"4c61bb46-57bd-4a99-9ea6-ac5a98302d73","Type":"ContainerStarted","Data":"434d4611ef8e1a0eec43e0ffcaa831bc9899c8d948ed66a1feb60c7073aaec23"}
Oct 30 21:46:04.738108 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:04.738074    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:46:04.738441 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:04.738169    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:46:05.277957 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:05.277922    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h" event={"ID":"4c61bb46-57bd-4a99-9ea6-ac5a98302d73","Type":"ContainerStarted","Data":"98f0716668449c349e33edf19623401201cae9da8d40193f393ab3fe239647b6"}
Oct 30 21:46:05.278129 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:05.278086    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:05.279710 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:05.279682    4553 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-xpm8h container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:46:05.279838 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:05.279731    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h" podUID="4c61bb46-57bd-4a99-9ea6-ac5a98302d73" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:46:05.488644 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:05.488585    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h" podStartSLOduration=1.639672657 podStartE2EDuration="5.48857141s" totalImagesPullingTime="3.848898753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:05.487696625 +0000 UTC m=+216.222059672" watchObservedRunningTime="2026-10-30 21:46:05.48857141 +0000 UTC m=+216.222934440"
Oct 30 21:46:06.282103 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:46:06.282067    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:46:28.199646 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:46:28.199609    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:46:58.201555 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:46:58.201515    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:47:05.689351 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.689314    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"]
Oct 30 21:47:05.693053 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.693022    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.739936 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.739897    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"]
Oct 30 21:47:05.785739 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.785705    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:47:05.785933 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.785803    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:47:05.785933 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.785852    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:47:05.785933 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.785873    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-7pn2f\""
Oct 30 21:47:05.789221 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.789201    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b8645af-284c-43bd-8916-c0fabaa4bbb5-metrics-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.789292 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.789234    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b8645af-284c-43bd-8916-c0fabaa4bbb5-webhook-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.789292 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.789272    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b8645af-284c-43bd-8916-c0fabaa4bbb5-tmp\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.789376 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.789364    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b8645af-284c-43bd-8916-c0fabaa4bbb5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.789445 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.789422    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpgwb\" (UniqueName: \"kubernetes.io/projected/7b8645af-284c-43bd-8916-c0fabaa4bbb5-kube-api-access-tpgwb\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.789513 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.789502    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b8645af-284c-43bd-8916-c0fabaa4bbb5-manifests\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.811635 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.811607    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:47:05.811635 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.811613    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:47:05.890671 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.890632    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b8645af-284c-43bd-8916-c0fabaa4bbb5-metrics-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.890671 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.890674    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b8645af-284c-43bd-8916-c0fabaa4bbb5-webhook-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.890944 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.890797    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b8645af-284c-43bd-8916-c0fabaa4bbb5-tmp\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.890944 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.890857    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b8645af-284c-43bd-8916-c0fabaa4bbb5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.890944 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.890875    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpgwb\" (UniqueName: \"kubernetes.io/projected/7b8645af-284c-43bd-8916-c0fabaa4bbb5-kube-api-access-tpgwb\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.890944 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.890916    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b8645af-284c-43bd-8916-c0fabaa4bbb5-manifests\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.891359 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.891326    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b8645af-284c-43bd-8916-c0fabaa4bbb5-manifests\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.893190 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.893162    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b8645af-284c-43bd-8916-c0fabaa4bbb5-tmp\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.893506 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.893484    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b8645af-284c-43bd-8916-c0fabaa4bbb5-webhook-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.893626 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.893605    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b8645af-284c-43bd-8916-c0fabaa4bbb5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.893668 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.893605    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b8645af-284c-43bd-8916-c0fabaa4bbb5-metrics-cert\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:05.950213 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:05.950121    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpgwb\" (UniqueName: \"kubernetes.io/projected/7b8645af-284c-43bd-8916-c0fabaa4bbb5-kube-api-access-tpgwb\") pod \"opendatahub-operator-controller-manager-578f754bb8-b8d6c\" (UID: \"7b8645af-284c-43bd-8916-c0fabaa4bbb5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:06.009288 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:06.009249    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:06.585233 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:06.585190    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"]
Oct 30 21:47:06.638025 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:47:06.637993    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8645af_284c_43bd_8916_c0fabaa4bbb5.slice/crio-1d9e1c57ebdbd90a39f30eb9ba4e72716db20b154d6e7b3f912015bfee847ebe WatchSource:0}: Error finding container 1d9e1c57ebdbd90a39f30eb9ba4e72716db20b154d6e7b3f912015bfee847ebe: Status 404 returned error can't find the container with id 1d9e1c57ebdbd90a39f30eb9ba4e72716db20b154d6e7b3f912015bfee847ebe
Oct 30 21:47:07.506054 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:07.506014    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c" event={"ID":"7b8645af-284c-43bd-8916-c0fabaa4bbb5","Type":"ContainerStarted","Data":"1d9e1c57ebdbd90a39f30eb9ba4e72716db20b154d6e7b3f912015bfee847ebe"}
Oct 30 21:47:09.189967 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:09.189933    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-fsqhl"]
Oct 30 21:47:09.194536 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:09.194508    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-fsqhl"
Oct 30 21:47:09.218787 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:09.218750    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvz5j\" (UniqueName: \"kubernetes.io/projected/0c60ffbb-13d8-4b1f-a8c1-a168f53e739f-kube-api-access-lvz5j\") pod \"cert-manager-597b95cb84-fsqhl\" (UID: \"0c60ffbb-13d8-4b1f-a8c1-a168f53e739f\") " pod="cert-manager/cert-manager-597b95cb84-fsqhl"
Oct 30 21:47:09.218957 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:09.218913    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c60ffbb-13d8-4b1f-a8c1-a168f53e739f-bound-sa-token\") pod \"cert-manager-597b95cb84-fsqhl\" (UID: \"0c60ffbb-13d8-4b1f-a8c1-a168f53e739f\") " pod="cert-manager/cert-manager-597b95cb84-fsqhl"
Oct 30 21:47:09.249019 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:09.248989    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 21:47:09.300701 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:09.300664    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5xzsz\""
Oct 30 21:47:09.300884 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:09.300727    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 21:47:09.300884 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:09.300747    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-fsqhl"]
Oct 30 21:47:09.320125 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:09.320088    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvz5j\" (UniqueName: \"kubernetes.io/projected/0c60ffbb-13d8-4b1f-a8c1-a168f53e739f-kube-api-access-lvz5j\") pod \"cert-manager-597b95cb84-fsqhl\" (UID: \"0c60ffbb-13d8-4b1f-a8c1-a168f53e739f\") " pod="cert-manager/cert-manager-597b95cb84-fsqhl"
Oct 30 21:47:09.320351 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:09.320212    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c60ffbb-13d8-4b1f-a8c1-a168f53e739f-bound-sa-token\") pod \"cert-manager-597b95cb84-fsqhl\" (UID: \"0c60ffbb-13d8-4b1f-a8c1-a168f53e739f\") " pod="cert-manager/cert-manager-597b95cb84-fsqhl"
Oct 30 21:47:09.383974 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:09.383934    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c60ffbb-13d8-4b1f-a8c1-a168f53e739f-bound-sa-token\") pod \"cert-manager-597b95cb84-fsqhl\" (UID: \"0c60ffbb-13d8-4b1f-a8c1-a168f53e739f\") " pod="cert-manager/cert-manager-597b95cb84-fsqhl"
Oct 30 21:47:09.384538 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:09.384501    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvz5j\" (UniqueName: \"kubernetes.io/projected/0c60ffbb-13d8-4b1f-a8c1-a168f53e739f-kube-api-access-lvz5j\") pod \"cert-manager-597b95cb84-fsqhl\" (UID: \"0c60ffbb-13d8-4b1f-a8c1-a168f53e739f\") " pod="cert-manager/cert-manager-597b95cb84-fsqhl"
Oct 30 21:47:09.525865 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:09.525771    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-fsqhl"
Oct 30 21:47:09.934533 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:09.934444    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-fsqhl"]
Oct 30 21:47:09.937925 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:47:09.937891    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c60ffbb_13d8_4b1f_a8c1_a168f53e739f.slice/crio-b293fb0bc247f89992214d118242d7b4799b55d101c7a8b2f31514c04a4d0ca1 WatchSource:0}: Error finding container b293fb0bc247f89992214d118242d7b4799b55d101c7a8b2f31514c04a4d0ca1: Status 404 returned error can't find the container with id b293fb0bc247f89992214d118242d7b4799b55d101c7a8b2f31514c04a4d0ca1
Oct 30 21:47:10.520673 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:10.520634    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c" event={"ID":"7b8645af-284c-43bd-8916-c0fabaa4bbb5","Type":"ContainerStarted","Data":"8a739e5a6013651685b285373750b5b6b0ef15442c7c975accc669f017e0599b"}
Oct 30 21:47:10.521843 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:10.521819    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-fsqhl" event={"ID":"0c60ffbb-13d8-4b1f-a8c1-a168f53e739f","Type":"ContainerStarted","Data":"b293fb0bc247f89992214d118242d7b4799b55d101c7a8b2f31514c04a4d0ca1"}
Oct 30 21:47:11.527895 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:11.527864    4553 generic.go:356] "Generic (PLEG): container finished" podID="7b8645af-284c-43bd-8916-c0fabaa4bbb5" containerID="8a739e5a6013651685b285373750b5b6b0ef15442c7c975accc669f017e0599b" exitCode=0
Oct 30 21:47:11.528334 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:11.527902    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c" event={"ID":"7b8645af-284c-43bd-8916-c0fabaa4bbb5","Type":"ContainerDied","Data":"8a739e5a6013651685b285373750b5b6b0ef15442c7c975accc669f017e0599b"}
Oct 30 21:47:13.539752 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:13.539716    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c" event={"ID":"7b8645af-284c-43bd-8916-c0fabaa4bbb5","Type":"ContainerStarted","Data":"d96df409c6ead474ceb2e6a2a8c562046ebab410136824e05f15f49f784fc6c9"}
Oct 30 21:47:13.540263 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:13.539811    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:13.827553 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:13.827435    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c" podStartSLOduration=3.498647339 podStartE2EDuration="8.827415963s" totalImagesPullingTime="4.851024152s" totalInitContainerRuntime="477.744472ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:13.775963383 +0000 UTC m=+284.510326428" watchObservedRunningTime="2026-10-30 21:47:13.827415963 +0000 UTC m=+284.561779098"
Oct 30 21:47:15.550338 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:15.550294    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-fsqhl" event={"ID":"0c60ffbb-13d8-4b1f-a8c1-a168f53e739f","Type":"ContainerStarted","Data":"b29d43ab120af36bdce0c041db1d153d60b1a80d25cc25f62b673d5f2ae2af10"}
Oct 30 21:47:15.665112 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:15.665058    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-fsqhl" podStartSLOduration=1.527803413 podStartE2EDuration="6.665045608s" totalImagesPullingTime="5.137242195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:15.664362077 +0000 UTC m=+286.398725109" watchObservedRunningTime="2026-10-30 21:47:15.665045608 +0000 UTC m=+286.399408638"
Oct 30 21:47:24.547055 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:24.547023    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-578f754bb8-b8d6c"
Oct 30 21:47:28.200518 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:47:28.200476    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:47:29.408262 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:29.408228    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 21:47:29.408781 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:29.408730    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 21:47:35.569021 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.568991    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-76f5684988-x9n4b"]
Oct 30 21:47:35.579410 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.579385    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76f5684988-x9n4b"
Oct 30 21:47:35.631924 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.631887    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 21:47:35.632100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.631986    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vrvww\""
Oct 30 21:47:35.660751 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.660715    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqqhx\" (UniqueName: \"kubernetes.io/projected/9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71-kube-api-access-qqqhx\") pod \"kube-auth-proxy-76f5684988-x9n4b\" (UID: \"9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-x9n4b"
Oct 30 21:47:35.660928 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.660781    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71-tls-certs\") pod \"kube-auth-proxy-76f5684988-x9n4b\" (UID: \"9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-x9n4b"
Oct 30 21:47:35.660928 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.660893    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71-tmp\") pod \"kube-auth-proxy-76f5684988-x9n4b\" (UID: \"9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-x9n4b"
Oct 30 21:47:35.682457 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.682422    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 21:47:35.683672 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.683652    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76f5684988-x9n4b"]
Oct 30 21:47:35.762251 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.762215    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71-tmp\") pod \"kube-auth-proxy-76f5684988-x9n4b\" (UID: \"9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-x9n4b"
Oct 30 21:47:35.762466 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.762288    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqhx\" (UniqueName: \"kubernetes.io/projected/9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71-kube-api-access-qqqhx\") pod \"kube-auth-proxy-76f5684988-x9n4b\" (UID: \"9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-x9n4b"
Oct 30 21:47:35.762466 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.762320    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71-tls-certs\") pod \"kube-auth-proxy-76f5684988-x9n4b\" (UID: \"9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-x9n4b"
Oct 30 21:47:35.764943 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.764907    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71-tmp\") pod \"kube-auth-proxy-76f5684988-x9n4b\" (UID: \"9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-x9n4b"
Oct 30 21:47:35.765081 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.765063    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71-tls-certs\") pod \"kube-auth-proxy-76f5684988-x9n4b\" (UID: \"9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-x9n4b"
Oct 30 21:47:35.819778 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.819709    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqhx\" (UniqueName: \"kubernetes.io/projected/9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71-kube-api-access-qqqhx\") pod \"kube-auth-proxy-76f5684988-x9n4b\" (UID: \"9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71\") " pod="openshift-ingress/kube-auth-proxy-76f5684988-x9n4b"
Oct 30 21:47:35.891114 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:35.891079    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76f5684988-x9n4b"
Oct 30 21:47:36.263644 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:36.263610    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76f5684988-x9n4b"]
Oct 30 21:47:36.267034 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:47:36.267008    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9564ef8e_5d55_4ece_a63f_d5d3b6bc3a71.slice/crio-ac669d2b4ef873ca985e34aefb30668537ab933d6f3c0d8ff234de667f1a8847 WatchSource:0}: Error finding container ac669d2b4ef873ca985e34aefb30668537ab933d6f3c0d8ff234de667f1a8847: Status 404 returned error can't find the container with id ac669d2b4ef873ca985e34aefb30668537ab933d6f3c0d8ff234de667f1a8847
Oct 30 21:47:36.268961 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:36.268943    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:47:36.636661 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:36.636620    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76f5684988-x9n4b" event={"ID":"9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71","Type":"ContainerStarted","Data":"ac669d2b4ef873ca985e34aefb30668537ab933d6f3c0d8ff234de667f1a8847"}
Oct 30 21:47:39.651216 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:39.651182    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76f5684988-x9n4b" event={"ID":"9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71","Type":"ContainerStarted","Data":"60402efbe925060f9c70610756107470fea21edd261671fbf9b3367132477178"}
Oct 30 21:47:39.768784 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:39.768721    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-76f5684988-x9n4b" podStartSLOduration=2.091142664 podStartE2EDuration="4.768703265s" totalImagesPullingTime="2.677560601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:39.768445684 +0000 UTC m=+310.502808716" watchObservedRunningTime="2026-10-30 21:47:39.768703265 +0000 UTC m=+310.503066298"
Oct 30 21:47:40.276535 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.276497    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r74s8"]
Oct 30 21:47:40.278629 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.278613    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8"
Oct 30 21:47:40.331876 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.331837    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 21:47:40.331876 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.331843    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:47:40.382508 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.382465    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:47:40.382665 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.382546    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hqkzq\""
Oct 30 21:47:40.388372 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.384790    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r74s8"]
Oct 30 21:47:40.399525 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.399491    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrr49\" (UniqueName: \"kubernetes.io/projected/bc861f92-6a61-4bcc-99fa-a513d7c75a64-kube-api-access-xrr49\") pod \"limitador-limitador-74dfcfb894-r74s8\" (UID: \"bc861f92-6a61-4bcc-99fa-a513d7c75a64\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8"
Oct 30 21:47:40.399692 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.399612    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc861f92-6a61-4bcc-99fa-a513d7c75a64-config-file\") pod \"limitador-limitador-74dfcfb894-r74s8\" (UID: \"bc861f92-6a61-4bcc-99fa-a513d7c75a64\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8"
Oct 30 21:47:40.500732 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.500700    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc861f92-6a61-4bcc-99fa-a513d7c75a64-config-file\") pod \"limitador-limitador-74dfcfb894-r74s8\" (UID: \"bc861f92-6a61-4bcc-99fa-a513d7c75a64\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8"
Oct 30 21:47:40.500926 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.500764    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrr49\" (UniqueName: \"kubernetes.io/projected/bc861f92-6a61-4bcc-99fa-a513d7c75a64-kube-api-access-xrr49\") pod \"limitador-limitador-74dfcfb894-r74s8\" (UID: \"bc861f92-6a61-4bcc-99fa-a513d7c75a64\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8"
Oct 30 21:47:40.501416 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.501381    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc861f92-6a61-4bcc-99fa-a513d7c75a64-config-file\") pod \"limitador-limitador-74dfcfb894-r74s8\" (UID: \"bc861f92-6a61-4bcc-99fa-a513d7c75a64\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8"
Oct 30 21:47:40.517625 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.517597    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r74s8"]
Oct 30 21:47:40.558567 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.558498    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrr49\" (UniqueName: \"kubernetes.io/projected/bc861f92-6a61-4bcc-99fa-a513d7c75a64-kube-api-access-xrr49\") pod \"limitador-limitador-74dfcfb894-r74s8\" (UID: \"bc861f92-6a61-4bcc-99fa-a513d7c75a64\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8"
Oct 30 21:47:40.590381 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.590347    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8"
Oct 30 21:47:40.964887 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:40.964859    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r74s8"]
Oct 30 21:47:40.967306 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:47:40.967271    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc861f92_6a61_4bcc_99fa_a513d7c75a64.slice/crio-001be8fab82ad14e0b655a811419bc4c4f19b05e7902882fbe58f65939717a6e WatchSource:0}: Error finding container 001be8fab82ad14e0b655a811419bc4c4f19b05e7902882fbe58f65939717a6e: Status 404 returned error can't find the container with id 001be8fab82ad14e0b655a811419bc4c4f19b05e7902882fbe58f65939717a6e
Oct 30 21:47:41.660771 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:41.660739    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8" event={"ID":"bc861f92-6a61-4bcc-99fa-a513d7c75a64","Type":"ContainerStarted","Data":"001be8fab82ad14e0b655a811419bc4c4f19b05e7902882fbe58f65939717a6e"}
Oct 30 21:47:44.674955 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:44.674859    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8" event={"ID":"bc861f92-6a61-4bcc-99fa-a513d7c75a64","Type":"ContainerStarted","Data":"de6b07d38372107f65a4c234d1dfc9f0ad69b07f5cc15e4ee072f12752af4d32"}
Oct 30 21:47:44.675389 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:44.674953    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8"
Oct 30 21:47:44.791356 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:44.791291    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8" podStartSLOduration=1.350196876 podStartE2EDuration="4.791276774s" totalImagesPullingTime="3.441079898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:44.789544314 +0000 UTC m=+315.523907344" watchObservedRunningTime="2026-10-30 21:47:44.791276774 +0000 UTC m=+315.525639806"
Oct 30 21:47:47.395597 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.395562    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"]
Oct 30 21:47:47.397853 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.397837    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:47:47.453336 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.453306    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-7sbkz\""
Oct 30 21:47:47.453518 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.453500    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:47:47.453570 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.453500    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:47:47.464944 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.464911    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tl7r\" (UniqueName: \"kubernetes.io/projected/63b50514-3b83-44bd-8f3a-6d7d947944c0-kube-api-access-4tl7r\") pod \"kserve-module-controller-manager-7566cf5fc8-fhwfv\" (UID: \"63b50514-3b83-44bd-8f3a-6d7d947944c0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:47:47.465198 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.464976    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63b50514-3b83-44bd-8f3a-6d7d947944c0-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-fhwfv\" (UID: \"63b50514-3b83-44bd-8f3a-6d7d947944c0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:47:47.465198 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.465024    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63b50514-3b83-44bd-8f3a-6d7d947944c0-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-fhwfv\" (UID: \"63b50514-3b83-44bd-8f3a-6d7d947944c0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:47:47.465198 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.465163    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63b50514-3b83-44bd-8f3a-6d7d947944c0-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-fhwfv\" (UID: \"63b50514-3b83-44bd-8f3a-6d7d947944c0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:47:47.555069 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.555029    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 21:47:47.556127 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.556101    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"]
Oct 30 21:47:47.566291 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.566268    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tl7r\" (UniqueName: \"kubernetes.io/projected/63b50514-3b83-44bd-8f3a-6d7d947944c0-kube-api-access-4tl7r\") pod \"kserve-module-controller-manager-7566cf5fc8-fhwfv\" (UID: \"63b50514-3b83-44bd-8f3a-6d7d947944c0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:47:47.566410 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.566321    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63b50514-3b83-44bd-8f3a-6d7d947944c0-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-fhwfv\" (UID: \"63b50514-3b83-44bd-8f3a-6d7d947944c0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:47:47.566410 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.566371    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63b50514-3b83-44bd-8f3a-6d7d947944c0-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-fhwfv\" (UID: \"63b50514-3b83-44bd-8f3a-6d7d947944c0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:47:47.566410 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.566397    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63b50514-3b83-44bd-8f3a-6d7d947944c0-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-fhwfv\" (UID: \"63b50514-3b83-44bd-8f3a-6d7d947944c0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:47:47.566757 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.566736    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63b50514-3b83-44bd-8f3a-6d7d947944c0-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-fhwfv\" (UID: \"63b50514-3b83-44bd-8f3a-6d7d947944c0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:47:47.566848 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.566773    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63b50514-3b83-44bd-8f3a-6d7d947944c0-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-fhwfv\" (UID: \"63b50514-3b83-44bd-8f3a-6d7d947944c0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:47:47.567366 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.567344    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/63b50514-3b83-44bd-8f3a-6d7d947944c0-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-fhwfv\" (UID: \"63b50514-3b83-44bd-8f3a-6d7d947944c0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:47:47.625298 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.625269    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tl7r\" (UniqueName: \"kubernetes.io/projected/63b50514-3b83-44bd-8f3a-6d7d947944c0-kube-api-access-4tl7r\") pod \"kserve-module-controller-manager-7566cf5fc8-fhwfv\" (UID: \"63b50514-3b83-44bd-8f3a-6d7d947944c0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:47:47.708813 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:47.708724    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:47:48.180178 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:48.180128    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"]
Oct 30 21:47:48.183163 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:47:48.183118    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b50514_3b83_44bd_8f3a_6d7d947944c0.slice/crio-fab7392992002c99e6fead8a9623d6805b180f2ab4872721295398df16866de0 WatchSource:0}: Error finding container fab7392992002c99e6fead8a9623d6805b180f2ab4872721295398df16866de0: Status 404 returned error can't find the container with id fab7392992002c99e6fead8a9623d6805b180f2ab4872721295398df16866de0
Oct 30 21:47:48.690796 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:48.690764    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv" event={"ID":"63b50514-3b83-44bd-8f3a-6d7d947944c0","Type":"ContainerStarted","Data":"fab7392992002c99e6fead8a9623d6805b180f2ab4872721295398df16866de0"}
Oct 30 21:47:50.704620 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:50.704574    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv" event={"ID":"63b50514-3b83-44bd-8f3a-6d7d947944c0","Type":"ContainerStarted","Data":"e2c34de76dbe278dbcaea129e1b018e8003323eb42e150b66f60950b8db1d1fd"}
Oct 30 21:47:50.705293 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:50.705269    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:47:50.876721 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:50.876662    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv" podStartSLOduration=2.197360084 podStartE2EDuration="3.876648237s" totalImagesPullingTime="1.679288153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:50.875689731 +0000 UTC m=+321.610052763" watchObservedRunningTime="2026-10-30 21:47:50.876648237 +0000 UTC m=+321.611011301"
Oct 30 21:47:55.680115 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:55.680082    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8"
Oct 30 21:47:57.654206 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:57.654170    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r74s8"]
Oct 30 21:47:57.654647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:57.654378    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8" podUID="bc861f92-6a61-4bcc-99fa-a513d7c75a64" containerName="limitador" containerID="cri-o://de6b07d38372107f65a4c234d1dfc9f0ad69b07f5cc15e4ee072f12752af4d32" gracePeriod=30
Oct 30 21:47:58.201242 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:47:58.201211    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:47:58.203899 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.203878    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8"
Oct 30 21:47:58.262582 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.262514    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc861f92-6a61-4bcc-99fa-a513d7c75a64-kube-api-access-xrr49\" (UniqueName: \"kubernetes.io/projected/bc861f92-6a61-4bcc-99fa-a513d7c75a64-kube-api-access-xrr49\") pod \"bc861f92-6a61-4bcc-99fa-a513d7c75a64\" (UID: \"bc861f92-6a61-4bcc-99fa-a513d7c75a64\") "
Oct 30 21:47:58.262699 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.262601    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc861f92-6a61-4bcc-99fa-a513d7c75a64-config-file\" (UniqueName: \"kubernetes.io/configmap/bc861f92-6a61-4bcc-99fa-a513d7c75a64-config-file\") pod \"bc861f92-6a61-4bcc-99fa-a513d7c75a64\" (UID: \"bc861f92-6a61-4bcc-99fa-a513d7c75a64\") "
Oct 30 21:47:58.262945 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.262919    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc861f92-6a61-4bcc-99fa-a513d7c75a64-config-file" pod "bc861f92-6a61-4bcc-99fa-a513d7c75a64" (UID: "bc861f92-6a61-4bcc-99fa-a513d7c75a64"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:47:58.264969 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.264951    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc861f92-6a61-4bcc-99fa-a513d7c75a64-kube-api-access-xrr49" pod "bc861f92-6a61-4bcc-99fa-a513d7c75a64" (UID: "bc861f92-6a61-4bcc-99fa-a513d7c75a64"). InnerVolumeSpecName "kube-api-access-xrr49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:47:58.363881 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.363839    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrr49\" (UniqueName: \"kubernetes.io/projected/bc861f92-6a61-4bcc-99fa-a513d7c75a64-kube-api-access-xrr49\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:47:58.363881 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.363878    4553 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bc861f92-6a61-4bcc-99fa-a513d7c75a64-config-file\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:47:58.738883 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.738846    4553 generic.go:356] "Generic (PLEG): container finished" podID="bc861f92-6a61-4bcc-99fa-a513d7c75a64" containerID="de6b07d38372107f65a4c234d1dfc9f0ad69b07f5cc15e4ee072f12752af4d32" exitCode=0
Oct 30 21:47:58.739376 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.738894    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8"
Oct 30 21:47:58.739376 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.738917    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8" event={"ID":"bc861f92-6a61-4bcc-99fa-a513d7c75a64","Type":"ContainerDied","Data":"de6b07d38372107f65a4c234d1dfc9f0ad69b07f5cc15e4ee072f12752af4d32"}
Oct 30 21:47:58.739376 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.738960    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-r74s8" event={"ID":"bc861f92-6a61-4bcc-99fa-a513d7c75a64","Type":"ContainerDied","Data":"001be8fab82ad14e0b655a811419bc4c4f19b05e7902882fbe58f65939717a6e"}
Oct 30 21:47:58.739376 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.738980    4553 scope.go:122] "RemoveContainer" containerID="de6b07d38372107f65a4c234d1dfc9f0ad69b07f5cc15e4ee072f12752af4d32"
Oct 30 21:47:58.747586 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.747567    4553 scope.go:122] "RemoveContainer" containerID="de6b07d38372107f65a4c234d1dfc9f0ad69b07f5cc15e4ee072f12752af4d32"
Oct 30 21:47:58.747842 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:47:58.747822    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6b07d38372107f65a4c234d1dfc9f0ad69b07f5cc15e4ee072f12752af4d32\": container with ID starting with de6b07d38372107f65a4c234d1dfc9f0ad69b07f5cc15e4ee072f12752af4d32 not found: ID does not exist" containerID="de6b07d38372107f65a4c234d1dfc9f0ad69b07f5cc15e4ee072f12752af4d32"
Oct 30 21:47:58.747902 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.747855    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6b07d38372107f65a4c234d1dfc9f0ad69b07f5cc15e4ee072f12752af4d32"} err="failed to get container status \"de6b07d38372107f65a4c234d1dfc9f0ad69b07f5cc15e4ee072f12752af4d32\": rpc error: code = NotFound desc = could not find container \"de6b07d38372107f65a4c234d1dfc9f0ad69b07f5cc15e4ee072f12752af4d32\": container with ID starting with de6b07d38372107f65a4c234d1dfc9f0ad69b07f5cc15e4ee072f12752af4d32 not found: ID does not exist"
Oct 30 21:47:58.904183 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.904129    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r74s8"]
Oct 30 21:47:58.954363 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:58.954334    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r74s8"]
Oct 30 21:47:59.390372 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:47:59.390339    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc861f92-6a61-4bcc-99fa-a513d7c75a64" path="/var/lib/kubelet/pods/bc861f92-6a61-4bcc-99fa-a513d7c75a64/volumes"
Oct 30 21:48:06.304597 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.304569    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9"]
Oct 30 21:48:06.305005 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.304878    4553 cpu_manager.go:414] "containerMap: removing container" podUID="bc861f92-6a61-4bcc-99fa-a513d7c75a64" containerName="limitador"
Oct 30 21:48:06.305005 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.304887    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc861f92-6a61-4bcc-99fa-a513d7c75a64" containerName="limitador"
Oct 30 21:48:06.305005 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.304953    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc861f92-6a61-4bcc-99fa-a513d7c75a64" containerName="limitador"
Oct 30 21:48:06.309253 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.309232    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9"
Oct 30 21:48:06.368278 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.368246    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9"]
Oct 30 21:48:06.417072 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.417044    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 21:48:06.417293 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.417097    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-mbd8s\""
Oct 30 21:48:06.417293 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.417120    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:48:06.417293 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.417167    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:48:06.434941 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.434873    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31ee7330-61f5-41ea-967f-23691becb52e-data\") pod \"postgres-7cdbf6dc44-nkpq9\" (UID: \"31ee7330-61f5-41ea-967f-23691becb52e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9"
Oct 30 21:48:06.435099 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.434978    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwxbp\" (UniqueName: \"kubernetes.io/projected/31ee7330-61f5-41ea-967f-23691becb52e-kube-api-access-bwxbp\") pod \"postgres-7cdbf6dc44-nkpq9\" (UID: \"31ee7330-61f5-41ea-967f-23691becb52e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9"
Oct 30 21:48:06.536060 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.536015    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31ee7330-61f5-41ea-967f-23691becb52e-data\") pod \"postgres-7cdbf6dc44-nkpq9\" (UID: \"31ee7330-61f5-41ea-967f-23691becb52e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9"
Oct 30 21:48:06.536281 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.536077    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxbp\" (UniqueName: \"kubernetes.io/projected/31ee7330-61f5-41ea-967f-23691becb52e-kube-api-access-bwxbp\") pod \"postgres-7cdbf6dc44-nkpq9\" (UID: \"31ee7330-61f5-41ea-967f-23691becb52e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9"
Oct 30 21:48:06.536500 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.536477    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/31ee7330-61f5-41ea-967f-23691becb52e-data\") pod \"postgres-7cdbf6dc44-nkpq9\" (UID: \"31ee7330-61f5-41ea-967f-23691becb52e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9"
Oct 30 21:48:06.594746 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.594717    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwxbp\" (UniqueName: \"kubernetes.io/projected/31ee7330-61f5-41ea-967f-23691becb52e-kube-api-access-bwxbp\") pod \"postgres-7cdbf6dc44-nkpq9\" (UID: \"31ee7330-61f5-41ea-967f-23691becb52e\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9"
Oct 30 21:48:06.622194 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.622162    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9"
Oct 30 21:48:06.994652 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:06.994625    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9"]
Oct 30 21:48:06.997374 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:48:06.997345    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ee7330_61f5_41ea_967f_23691becb52e.slice/crio-0fa4ae619a4276670a8064e8cba02b490907938aa451ef6bc70ab298c431b918 WatchSource:0}: Error finding container 0fa4ae619a4276670a8064e8cba02b490907938aa451ef6bc70ab298c431b918: Status 404 returned error can't find the container with id 0fa4ae619a4276670a8064e8cba02b490907938aa451ef6bc70ab298c431b918
Oct 30 21:48:07.775986 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:07.775950    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9" event={"ID":"31ee7330-61f5-41ea-967f-23691becb52e","Type":"ContainerStarted","Data":"0fa4ae619a4276670a8064e8cba02b490907938aa451ef6bc70ab298c431b918"}
Oct 30 21:48:11.709876 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:11.709844    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-fhwfv"
Oct 30 21:48:14.812223 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:14.812187    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9" event={"ID":"31ee7330-61f5-41ea-967f-23691becb52e","Type":"ContainerStarted","Data":"e9117df0707e019e1cde91631b49dd10fa10aed9c1d022f7fb49734cca34fd94"}
Oct 30 21:48:14.812656 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:14.812341    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9"
Oct 30 21:48:14.931631 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:14.931577    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9" podStartSLOduration=1.604998227 podStartE2EDuration="8.931560726s" totalImagesPullingTime="7.326562499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:14.931070913 +0000 UTC m=+345.665433946" watchObservedRunningTime="2026-10-30 21:48:14.931560726 +0000 UTC m=+345.665923757"
Oct 30 21:48:20.846442 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:20.846406    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-nkpq9"
Oct 30 21:48:28.200098 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:48:28.200062    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:48:37.864726 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:37.864679    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65c8fd6c86-kxlpc"]
Oct 30 21:48:37.875105 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:37.875078    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65c8fd6c86-kxlpc"
Oct 30 21:48:37.922033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:37.921997    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbqwj\" (UniqueName: \"kubernetes.io/projected/e74add30-a162-4575-9bfd-54089e9c0e2f-kube-api-access-pbqwj\") pod \"maas-controller-65c8fd6c86-kxlpc\" (UID: \"e74add30-a162-4575-9bfd-54089e9c0e2f\") " pod="opendatahub/maas-controller-65c8fd6c86-kxlpc"
Oct 30 21:48:37.922237 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:37.922070    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e74add30-a162-4575-9bfd-54089e9c0e2f-webhook-cert\") pod \"maas-controller-65c8fd6c86-kxlpc\" (UID: \"e74add30-a162-4575-9bfd-54089e9c0e2f\") " pod="opendatahub/maas-controller-65c8fd6c86-kxlpc"
Oct 30 21:48:37.922237 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:37.922173    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e74add30-a162-4575-9bfd-54089e9c0e2f-metrics-certs\") pod \"maas-controller-65c8fd6c86-kxlpc\" (UID: \"e74add30-a162-4575-9bfd-54089e9c0e2f\") " pod="opendatahub/maas-controller-65c8fd6c86-kxlpc"
Oct 30 21:48:37.987961 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:37.987914    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 21:48:37.988181 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:37.988000    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 21:48:37.988181 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:37.988039    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dfwlx\""
Oct 30 21:48:37.988181 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:37.988086    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 21:48:37.989342 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:37.989319    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65c8fd6c86-kxlpc"]
Oct 30 21:48:38.023104 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.023064    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e74add30-a162-4575-9bfd-54089e9c0e2f-webhook-cert\") pod \"maas-controller-65c8fd6c86-kxlpc\" (UID: \"e74add30-a162-4575-9bfd-54089e9c0e2f\") " pod="opendatahub/maas-controller-65c8fd6c86-kxlpc"
Oct 30 21:48:38.023330 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.023125    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e74add30-a162-4575-9bfd-54089e9c0e2f-metrics-certs\") pod \"maas-controller-65c8fd6c86-kxlpc\" (UID: \"e74add30-a162-4575-9bfd-54089e9c0e2f\") " pod="opendatahub/maas-controller-65c8fd6c86-kxlpc"
Oct 30 21:48:38.023330 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.023200    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbqwj\" (UniqueName: \"kubernetes.io/projected/e74add30-a162-4575-9bfd-54089e9c0e2f-kube-api-access-pbqwj\") pod \"maas-controller-65c8fd6c86-kxlpc\" (UID: \"e74add30-a162-4575-9bfd-54089e9c0e2f\") " pod="opendatahub/maas-controller-65c8fd6c86-kxlpc"
Oct 30 21:48:38.025873 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.025840    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e74add30-a162-4575-9bfd-54089e9c0e2f-webhook-cert\") pod \"maas-controller-65c8fd6c86-kxlpc\" (UID: \"e74add30-a162-4575-9bfd-54089e9c0e2f\") " pod="opendatahub/maas-controller-65c8fd6c86-kxlpc"
Oct 30 21:48:38.025987 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.025840    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e74add30-a162-4575-9bfd-54089e9c0e2f-metrics-certs\") pod \"maas-controller-65c8fd6c86-kxlpc\" (UID: \"e74add30-a162-4575-9bfd-54089e9c0e2f\") " pod="opendatahub/maas-controller-65c8fd6c86-kxlpc"
Oct 30 21:48:38.084275 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.084246    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbqwj\" (UniqueName: \"kubernetes.io/projected/e74add30-a162-4575-9bfd-54089e9c0e2f-kube-api-access-pbqwj\") pod \"maas-controller-65c8fd6c86-kxlpc\" (UID: \"e74add30-a162-4575-9bfd-54089e9c0e2f\") " pod="opendatahub/maas-controller-65c8fd6c86-kxlpc"
Oct 30 21:48:38.188072 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.187985    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65c8fd6c86-kxlpc"
Oct 30 21:48:38.449317 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.449243    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-655bd8bb49-gwsxw"]
Oct 30 21:48:38.453688 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.453665    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-655bd8bb49-gwsxw"
Oct 30 21:48:38.511361 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.511332    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-655bd8bb49-gwsxw"]
Oct 30 21:48:38.527223 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.527196    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab0381a1-70ca-4b49-9e1e-92db62e53c40-metrics-certs\") pod \"maas-controller-655bd8bb49-gwsxw\" (UID: \"ab0381a1-70ca-4b49-9e1e-92db62e53c40\") " pod="opendatahub/maas-controller-655bd8bb49-gwsxw"
Oct 30 21:48:38.527413 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.527243    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab0381a1-70ca-4b49-9e1e-92db62e53c40-webhook-cert\") pod \"maas-controller-655bd8bb49-gwsxw\" (UID: \"ab0381a1-70ca-4b49-9e1e-92db62e53c40\") " pod="opendatahub/maas-controller-655bd8bb49-gwsxw"
Oct 30 21:48:38.527413 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.527281    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqtjc\" (UniqueName: \"kubernetes.io/projected/ab0381a1-70ca-4b49-9e1e-92db62e53c40-kube-api-access-xqtjc\") pod \"maas-controller-655bd8bb49-gwsxw\" (UID: \"ab0381a1-70ca-4b49-9e1e-92db62e53c40\") " pod="opendatahub/maas-controller-655bd8bb49-gwsxw"
Oct 30 21:48:38.628003 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.627960    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab0381a1-70ca-4b49-9e1e-92db62e53c40-metrics-certs\") pod \"maas-controller-655bd8bb49-gwsxw\" (UID: \"ab0381a1-70ca-4b49-9e1e-92db62e53c40\") " pod="opendatahub/maas-controller-655bd8bb49-gwsxw"
Oct 30 21:48:38.628241 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.628023    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab0381a1-70ca-4b49-9e1e-92db62e53c40-webhook-cert\") pod \"maas-controller-655bd8bb49-gwsxw\" (UID: \"ab0381a1-70ca-4b49-9e1e-92db62e53c40\") " pod="opendatahub/maas-controller-655bd8bb49-gwsxw"
Oct 30 21:48:38.628241 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.628070    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqtjc\" (UniqueName: \"kubernetes.io/projected/ab0381a1-70ca-4b49-9e1e-92db62e53c40-kube-api-access-xqtjc\") pod \"maas-controller-655bd8bb49-gwsxw\" (UID: \"ab0381a1-70ca-4b49-9e1e-92db62e53c40\") " pod="opendatahub/maas-controller-655bd8bb49-gwsxw"
Oct 30 21:48:38.630827 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.630800    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab0381a1-70ca-4b49-9e1e-92db62e53c40-metrics-certs\") pod \"maas-controller-655bd8bb49-gwsxw\" (UID: \"ab0381a1-70ca-4b49-9e1e-92db62e53c40\") " pod="opendatahub/maas-controller-655bd8bb49-gwsxw"
Oct 30 21:48:38.630944 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.630800    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ab0381a1-70ca-4b49-9e1e-92db62e53c40-webhook-cert\") pod \"maas-controller-655bd8bb49-gwsxw\" (UID: \"ab0381a1-70ca-4b49-9e1e-92db62e53c40\") " pod="opendatahub/maas-controller-655bd8bb49-gwsxw"
Oct 30 21:48:38.639549 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.639526    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65c8fd6c86-kxlpc"]
Oct 30 21:48:38.641673 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:48:38.641649    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74add30_a162_4575_9bfd_54089e9c0e2f.slice/crio-687fdf14a75b09a69aede5f4bd908892e5de65e9ed3d9635cef5dd05bfa4c50c WatchSource:0}: Error finding container 687fdf14a75b09a69aede5f4bd908892e5de65e9ed3d9635cef5dd05bfa4c50c: Status 404 returned error can't find the container with id 687fdf14a75b09a69aede5f4bd908892e5de65e9ed3d9635cef5dd05bfa4c50c
Oct 30 21:48:38.691868 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.691839    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqtjc\" (UniqueName: \"kubernetes.io/projected/ab0381a1-70ca-4b49-9e1e-92db62e53c40-kube-api-access-xqtjc\") pod \"maas-controller-655bd8bb49-gwsxw\" (UID: \"ab0381a1-70ca-4b49-9e1e-92db62e53c40\") " pod="opendatahub/maas-controller-655bd8bb49-gwsxw"
Oct 30 21:48:38.766817 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.766725    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-655bd8bb49-gwsxw"
Oct 30 21:48:38.909856 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:38.909827    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65c8fd6c86-kxlpc" event={"ID":"e74add30-a162-4575-9bfd-54089e9c0e2f","Type":"ContainerStarted","Data":"687fdf14a75b09a69aede5f4bd908892e5de65e9ed3d9635cef5dd05bfa4c50c"}
Oct 30 21:48:39.145273 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:39.145250    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-655bd8bb49-gwsxw"]
Oct 30 21:48:39.147302 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:48:39.147274    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0381a1_70ca_4b49_9e1e_92db62e53c40.slice/crio-486797a51d2992c215ffaebcec136b013b433f98465fc25101f480c82ffa05e6 WatchSource:0}: Error finding container 486797a51d2992c215ffaebcec136b013b433f98465fc25101f480c82ffa05e6: Status 404 returned error can't find the container with id 486797a51d2992c215ffaebcec136b013b433f98465fc25101f480c82ffa05e6
Oct 30 21:48:39.924157 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:39.919807    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-655bd8bb49-gwsxw" event={"ID":"ab0381a1-70ca-4b49-9e1e-92db62e53c40","Type":"ContainerStarted","Data":"486797a51d2992c215ffaebcec136b013b433f98465fc25101f480c82ffa05e6"}
Oct 30 21:48:45.954287 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:45.954250    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65c8fd6c86-kxlpc" event={"ID":"e74add30-a162-4575-9bfd-54089e9c0e2f","Type":"ContainerStarted","Data":"4737502214eb6357f1223ecd59419a1e7e0cf30b38afa30c671f85e313372ef0"}
Oct 30 21:48:45.954840 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:45.954351    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65c8fd6c86-kxlpc"
Oct 30 21:48:45.956276 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:45.956243    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-655bd8bb49-gwsxw" event={"ID":"ab0381a1-70ca-4b49-9e1e-92db62e53c40","Type":"ContainerStarted","Data":"d5c5818abe3a4830087d12ef003c13cad0a9af9358201294f8aa35e4b147ef35"}
Oct 30 21:48:45.956437 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:45.956420    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-655bd8bb49-gwsxw"
Oct 30 21:48:46.072917 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:46.072862    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-65c8fd6c86-kxlpc" podStartSLOduration=2.361669361 podStartE2EDuration="9.072846552s" totalImagesPullingTime="6.711177191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:46.072051836 +0000 UTC m=+376.806414956" watchObservedRunningTime="2026-10-30 21:48:46.072846552 +0000 UTC m=+376.807209582"
Oct 30 21:48:46.206194 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:46.206079    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-655bd8bb49-gwsxw" podStartSLOduration=2.311913766 podStartE2EDuration="8.20606282s" totalImagesPullingTime="5.894149054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:46.20398898 +0000 UTC m=+376.938352025" watchObservedRunningTime="2026-10-30 21:48:46.20606282 +0000 UTC m=+376.940425852"
Oct 30 21:48:47.937994 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:47.937959    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"]
Oct 30 21:48:47.943809 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:47.943783    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.042595 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.042564    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b459720-eccc-40bb-ad95-3032719cc96c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.042776 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.042620    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.042776 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.042701    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b459720-eccc-40bb-ad95-3032719cc96c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.042776 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.042734    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-844xm\" (UniqueName: \"kubernetes.io/projected/1b459720-eccc-40bb-ad95-3032719cc96c-kube-api-access-844xm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.042776 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.042765    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.042909 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.042869    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.042909 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.042888    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b459720-eccc-40bb-ad95-3032719cc96c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.055752 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.055719    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"]
Oct 30 21:48:48.143417 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.143363    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.143417 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.143423    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b459720-eccc-40bb-ad95-3032719cc96c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.143682 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.143449    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-844xm\" (UniqueName: \"kubernetes.io/projected/1b459720-eccc-40bb-ad95-3032719cc96c-kube-api-access-844xm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.143682 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.143480    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.143682 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.143519    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.143682 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.143543    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b459720-eccc-40bb-ad95-3032719cc96c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.143682 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.143607    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b459720-eccc-40bb-ad95-3032719cc96c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.144398 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.144372    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b459720-eccc-40bb-ad95-3032719cc96c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.145796 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.145766    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b459720-eccc-40bb-ad95-3032719cc96c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.145974 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.145957    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.146299 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.146276    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.146391 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.146342    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.204003 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.203930    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b459720-eccc-40bb-ad95-3032719cc96c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.204305 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.204281    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-844xm\" (UniqueName: \"kubernetes.io/projected/1b459720-eccc-40bb-ad95-3032719cc96c-kube-api-access-844xm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q5v68\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.254254 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.254222    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:48.679097 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.679063    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"]
Oct 30 21:48:48.682107 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:48:48.682079    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b459720_eccc_40bb_ad95_3032719cc96c.slice/crio-78211fbc6b61ec4fa851918335a5e6c5f9b193d855a85905bddda7d3236a8587 WatchSource:0}: Error finding container 78211fbc6b61ec4fa851918335a5e6c5f9b193d855a85905bddda7d3236a8587: Status 404 returned error can't find the container with id 78211fbc6b61ec4fa851918335a5e6c5f9b193d855a85905bddda7d3236a8587
Oct 30 21:48:48.684299 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.684257    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:48:48.684398 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.684321    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:48:48.971812 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.971719    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68" event={"ID":"1b459720-eccc-40bb-ad95-3032719cc96c","Type":"ContainerStarted","Data":"f9eacde1e10aece24887a50a4cbe5000ea0850af6d91cc7ae17a01b53b62dad0"}
Oct 30 21:48:48.971812 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.971782    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68" event={"ID":"1b459720-eccc-40bb-ad95-3032719cc96c","Type":"ContainerStarted","Data":"78211fbc6b61ec4fa851918335a5e6c5f9b193d855a85905bddda7d3236a8587"}
Oct 30 21:48:48.972294 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:48.971842    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:49.093763 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:49.093705    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68" podStartSLOduration=2.093692839 podStartE2EDuration="2.093692839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:49.092963674 +0000 UTC m=+379.827326715" watchObservedRunningTime="2026-10-30 21:48:49.093692839 +0000 UTC m=+379.828055869"
Oct 30 21:48:49.981650 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:49.981615    4553 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-q5v68 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:48:49.982265 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:49.982230    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68" podUID="1b459720-eccc-40bb-ad95-3032719cc96c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:48:52.980099 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:52.980071    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 21:48:53.174171 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.171730    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"]
Oct 30 21:48:53.174171 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.172035    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h" podUID="4c61bb46-57bd-4a99-9ea6-ac5a98302d73" containerName="discovery" containerID="cri-o://98f0716668449c349e33edf19623401201cae9da8d40193f393ab3fe239647b6" gracePeriod=30
Oct 30 21:48:53.425402 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.425373    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:48:53.594358 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.594324    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-csr-ca-configmap\") pod \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") "
Oct 30 21:48:53.594606 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.594409    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-token\" (UniqueName: \"kubernetes.io/projected/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-token\") pod \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") "
Oct 30 21:48:53.594606 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.594441    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-local-certs\") pod \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") "
Oct 30 21:48:53.594606 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.594468    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-csr-dns-cert\") pod \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") "
Oct 30 21:48:53.594606 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.594549    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-cacerts\" (UniqueName: \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-cacerts\") pod \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") "
Oct 30 21:48:53.594606 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.594597    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-kubeconfig\") pod \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") "
Oct 30 21:48:53.595269 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.594652    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-kube-api-access-vpmc7\" (UniqueName: \"kubernetes.io/projected/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-kube-api-access-vpmc7\") pod \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\" (UID: \"4c61bb46-57bd-4a99-9ea6-ac5a98302d73\") "
Oct 30 21:48:53.595637 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.595601    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-csr-ca-configmap" pod "4c61bb46-57bd-4a99-9ea6-ac5a98302d73" (UID: "4c61bb46-57bd-4a99-9ea6-ac5a98302d73"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:48:53.598011 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.597961    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-local-certs" pod "4c61bb46-57bd-4a99-9ea6-ac5a98302d73" (UID: "4c61bb46-57bd-4a99-9ea6-ac5a98302d73"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:48:53.598162 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.598059    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-cacerts" pod "4c61bb46-57bd-4a99-9ea6-ac5a98302d73" (UID: "4c61bb46-57bd-4a99-9ea6-ac5a98302d73"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:53.598261 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.598227    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-kubeconfig" pod "4c61bb46-57bd-4a99-9ea6-ac5a98302d73" (UID: "4c61bb46-57bd-4a99-9ea6-ac5a98302d73"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:53.598827 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.598799    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-csr-dns-cert" pod "4c61bb46-57bd-4a99-9ea6-ac5a98302d73" (UID: "4c61bb46-57bd-4a99-9ea6-ac5a98302d73"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:53.598936 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.598862    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-kube-api-access-vpmc7" pod "4c61bb46-57bd-4a99-9ea6-ac5a98302d73" (UID: "4c61bb46-57bd-4a99-9ea6-ac5a98302d73"). InnerVolumeSpecName "kube-api-access-vpmc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:53.599172 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.599121    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-token" pod "4c61bb46-57bd-4a99-9ea6-ac5a98302d73" (UID: "4c61bb46-57bd-4a99-9ea6-ac5a98302d73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:53.695529 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.695502    4553 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-cacerts\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:48:53.695529 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.695527    4553 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-kubeconfig\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:48:53.695529 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.695538    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpmc7\" (UniqueName: \"kubernetes.io/projected/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-kube-api-access-vpmc7\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:48:53.695762 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.695548    4553 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-csr-ca-configmap\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:48:53.695762 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.695556    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:48:53.695762 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.695565    4553 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-local-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:48:53.695762 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.695573    4553 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c61bb46-57bd-4a99-9ea6-ac5a98302d73-istio-csr-dns-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:48:53.996194 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.996089    4553 generic.go:356] "Generic (PLEG): container finished" podID="4c61bb46-57bd-4a99-9ea6-ac5a98302d73" containerID="98f0716668449c349e33edf19623401201cae9da8d40193f393ab3fe239647b6" exitCode=0
Oct 30 21:48:53.996194 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.996160    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"
Oct 30 21:48:53.996194 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.996178    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h" event={"ID":"4c61bb46-57bd-4a99-9ea6-ac5a98302d73","Type":"ContainerDied","Data":"98f0716668449c349e33edf19623401201cae9da8d40193f393ab3fe239647b6"}
Oct 30 21:48:53.996667 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.996237    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h" event={"ID":"4c61bb46-57bd-4a99-9ea6-ac5a98302d73","Type":"ContainerDied","Data":"434d4611ef8e1a0eec43e0ffcaa831bc9899c8d948ed66a1feb60c7073aaec23"}
Oct 30 21:48:53.996667 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:53.996266    4553 scope.go:122] "RemoveContainer" containerID="98f0716668449c349e33edf19623401201cae9da8d40193f393ab3fe239647b6"
Oct 30 21:48:54.005160 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:54.005124    4553 scope.go:122] "RemoveContainer" containerID="98f0716668449c349e33edf19623401201cae9da8d40193f393ab3fe239647b6"
Oct 30 21:48:54.005454 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:48:54.005437    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f0716668449c349e33edf19623401201cae9da8d40193f393ab3fe239647b6\": container with ID starting with 98f0716668449c349e33edf19623401201cae9da8d40193f393ab3fe239647b6 not found: ID does not exist" containerID="98f0716668449c349e33edf19623401201cae9da8d40193f393ab3fe239647b6"
Oct 30 21:48:54.005497 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:54.005467    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f0716668449c349e33edf19623401201cae9da8d40193f393ab3fe239647b6"} err="failed to get container status \"98f0716668449c349e33edf19623401201cae9da8d40193f393ab3fe239647b6\": rpc error: code = NotFound desc = could not find container \"98f0716668449c349e33edf19623401201cae9da8d40193f393ab3fe239647b6\": container with ID starting with 98f0716668449c349e33edf19623401201cae9da8d40193f393ab3fe239647b6 not found: ID does not exist"
Oct 30 21:48:54.162580 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:54.162549    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"]
Oct 30 21:48:54.165895 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:54.165868    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpm8h"]
Oct 30 21:48:55.390483 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:55.390451    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c61bb46-57bd-4a99-9ea6-ac5a98302d73" path="/var/lib/kubelet/pods/4c61bb46-57bd-4a99-9ea6-ac5a98302d73/volumes"
Oct 30 21:48:56.963825 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:56.963797    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65c8fd6c86-kxlpc"
Oct 30 21:48:56.964344 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:56.963873    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-655bd8bb49-gwsxw"
Oct 30 21:48:57.229968 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:57.229887    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65c8fd6c86-kxlpc"]
Oct 30 21:48:57.230127 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:57.230100    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-65c8fd6c86-kxlpc" podUID="e74add30-a162-4575-9bfd-54089e9c0e2f" containerName="manager" containerID="cri-o://4737502214eb6357f1223ecd59419a1e7e0cf30b38afa30c671f85e313372ef0" gracePeriod=30
Oct 30 21:48:57.473958 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:57.473935    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65c8fd6c86-kxlpc"
Oct 30 21:48:57.630911 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:57.630874    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e74add30-a162-4575-9bfd-54089e9c0e2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e74add30-a162-4575-9bfd-54089e9c0e2f-metrics-certs\") pod \"e74add30-a162-4575-9bfd-54089e9c0e2f\" (UID: \"e74add30-a162-4575-9bfd-54089e9c0e2f\") "
Oct 30 21:48:57.630911 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:57.630928    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e74add30-a162-4575-9bfd-54089e9c0e2f-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e74add30-a162-4575-9bfd-54089e9c0e2f-webhook-cert\") pod \"e74add30-a162-4575-9bfd-54089e9c0e2f\" (UID: \"e74add30-a162-4575-9bfd-54089e9c0e2f\") "
Oct 30 21:48:57.631276 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:57.630995    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74add30-a162-4575-9bfd-54089e9c0e2f-kube-api-access-pbqwj\" (UniqueName: \"kubernetes.io/projected/e74add30-a162-4575-9bfd-54089e9c0e2f-kube-api-access-pbqwj\") pod \"e74add30-a162-4575-9bfd-54089e9c0e2f\" (UID: \"e74add30-a162-4575-9bfd-54089e9c0e2f\") "
Oct 30 21:48:57.633463 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:57.633416    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74add30-a162-4575-9bfd-54089e9c0e2f-webhook-cert" pod "e74add30-a162-4575-9bfd-54089e9c0e2f" (UID: "e74add30-a162-4575-9bfd-54089e9c0e2f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:57.633463 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:57.633435    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74add30-a162-4575-9bfd-54089e9c0e2f-metrics-certs" pod "e74add30-a162-4575-9bfd-54089e9c0e2f" (UID: "e74add30-a162-4575-9bfd-54089e9c0e2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:57.633621 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:57.633470    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74add30-a162-4575-9bfd-54089e9c0e2f-kube-api-access-pbqwj" pod "e74add30-a162-4575-9bfd-54089e9c0e2f" (UID: "e74add30-a162-4575-9bfd-54089e9c0e2f"). InnerVolumeSpecName "kube-api-access-pbqwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:57.732265 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:57.732225    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbqwj\" (UniqueName: \"kubernetes.io/projected/e74add30-a162-4575-9bfd-54089e9c0e2f-kube-api-access-pbqwj\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:48:57.732265 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:57.732258    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e74add30-a162-4575-9bfd-54089e9c0e2f-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:48:57.732265 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:57.732267    4553 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e74add30-a162-4575-9bfd-54089e9c0e2f-webhook-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:48:58.015155 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:58.015060    4553 generic.go:356] "Generic (PLEG): container finished" podID="e74add30-a162-4575-9bfd-54089e9c0e2f" containerID="4737502214eb6357f1223ecd59419a1e7e0cf30b38afa30c671f85e313372ef0" exitCode=0
Oct 30 21:48:58.015155 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:58.015110    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65c8fd6c86-kxlpc"
Oct 30 21:48:58.015613 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:58.015125    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65c8fd6c86-kxlpc" event={"ID":"e74add30-a162-4575-9bfd-54089e9c0e2f","Type":"ContainerDied","Data":"4737502214eb6357f1223ecd59419a1e7e0cf30b38afa30c671f85e313372ef0"}
Oct 30 21:48:58.015613 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:58.015218    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65c8fd6c86-kxlpc" event={"ID":"e74add30-a162-4575-9bfd-54089e9c0e2f","Type":"ContainerDied","Data":"687fdf14a75b09a69aede5f4bd908892e5de65e9ed3d9635cef5dd05bfa4c50c"}
Oct 30 21:48:58.015613 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:58.015249    4553 scope.go:122] "RemoveContainer" containerID="4737502214eb6357f1223ecd59419a1e7e0cf30b38afa30c671f85e313372ef0"
Oct 30 21:48:58.023952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:58.023934    4553 scope.go:122] "RemoveContainer" containerID="4737502214eb6357f1223ecd59419a1e7e0cf30b38afa30c671f85e313372ef0"
Oct 30 21:48:58.024229 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:48:58.024213    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4737502214eb6357f1223ecd59419a1e7e0cf30b38afa30c671f85e313372ef0\": container with ID starting with 4737502214eb6357f1223ecd59419a1e7e0cf30b38afa30c671f85e313372ef0 not found: ID does not exist" containerID="4737502214eb6357f1223ecd59419a1e7e0cf30b38afa30c671f85e313372ef0"
Oct 30 21:48:58.024340 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:58.024237    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4737502214eb6357f1223ecd59419a1e7e0cf30b38afa30c671f85e313372ef0"} err="failed to get container status \"4737502214eb6357f1223ecd59419a1e7e0cf30b38afa30c671f85e313372ef0\": rpc error: code = NotFound desc = could not find container \"4737502214eb6357f1223ecd59419a1e7e0cf30b38afa30c671f85e313372ef0\": container with ID starting with 4737502214eb6357f1223ecd59419a1e7e0cf30b38afa30c671f85e313372ef0 not found: ID does not exist"
Oct 30 21:48:58.182153 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:58.182099    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65c8fd6c86-kxlpc"]
Oct 30 21:48:58.192322 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:58.188558    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65c8fd6c86-kxlpc"]
Oct 30 21:48:58.203533 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:48:58.203491    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:48:59.390832 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:48:59.390799    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e74add30-a162-4575-9bfd-54089e9c0e2f" path="/var/lib/kubelet/pods/e74add30-a162-4575-9bfd-54089e9c0e2f/volumes"
Oct 30 21:49:22.374997 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.374915    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"]
Oct 30 21:49:22.375422 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.375308    4553 cpu_manager.go:414] "containerMap: removing container" podUID="e74add30-a162-4575-9bfd-54089e9c0e2f" containerName="manager"
Oct 30 21:49:22.375422 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.375321    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e74add30-a162-4575-9bfd-54089e9c0e2f" containerName="manager"
Oct 30 21:49:22.375422 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.375345    4553 cpu_manager.go:414] "containerMap: removing container" podUID="4c61bb46-57bd-4a99-9ea6-ac5a98302d73" containerName="discovery"
Oct 30 21:49:22.375422 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.375350    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c61bb46-57bd-4a99-9ea6-ac5a98302d73" containerName="discovery"
Oct 30 21:49:22.375422 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.375407    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="e74add30-a162-4575-9bfd-54089e9c0e2f" containerName="manager"
Oct 30 21:49:22.375422 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.375418    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c61bb46-57bd-4a99-9ea6-ac5a98302d73" containerName="discovery"
Oct 30 21:49:22.382695 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.382679    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:22.436953 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.436919    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"]
Oct 30 21:49:22.447247 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.447219    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aea20d56-947f-4b93-9352-4a361fb40961-metrics-certs\") pod \"maas-api-8c9d8544c-t7vr6\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") " pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:22.447399 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.447346    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aea20d56-947f-4b93-9352-4a361fb40961-maas-api-tls\") pod \"maas-api-8c9d8544c-t7vr6\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") " pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:22.447552 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.447527    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aea20d56-947f-4b93-9352-4a361fb40961-openshift-service-ca\") pod \"maas-api-8c9d8544c-t7vr6\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") " pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:22.447604 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.447573    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fcnz\" (UniqueName: \"kubernetes.io/projected/aea20d56-947f-4b93-9352-4a361fb40961-kube-api-access-7fcnz\") pod \"maas-api-8c9d8544c-t7vr6\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") " pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:22.485762 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.485727    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:49:22.486336 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.486319    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-489tq\""
Oct 30 21:49:22.486397 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.486334    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 21:49:22.548146 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.548108    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aea20d56-947f-4b93-9352-4a361fb40961-maas-api-tls\") pod \"maas-api-8c9d8544c-t7vr6\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") " pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:22.548345 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.548222    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aea20d56-947f-4b93-9352-4a361fb40961-openshift-service-ca\") pod \"maas-api-8c9d8544c-t7vr6\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") " pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:22.548345 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.548248    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fcnz\" (UniqueName: \"kubernetes.io/projected/aea20d56-947f-4b93-9352-4a361fb40961-kube-api-access-7fcnz\") pod \"maas-api-8c9d8544c-t7vr6\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") " pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:22.548345 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.548271    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aea20d56-947f-4b93-9352-4a361fb40961-metrics-certs\") pod \"maas-api-8c9d8544c-t7vr6\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") " pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:22.548936 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.548909    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aea20d56-947f-4b93-9352-4a361fb40961-openshift-service-ca\") pod \"maas-api-8c9d8544c-t7vr6\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") " pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:22.550828 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.550799    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aea20d56-947f-4b93-9352-4a361fb40961-metrics-certs\") pod \"maas-api-8c9d8544c-t7vr6\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") " pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:22.550950 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.550890    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aea20d56-947f-4b93-9352-4a361fb40961-maas-api-tls\") pod \"maas-api-8c9d8544c-t7vr6\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") " pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:22.606583 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.606550    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fcnz\" (UniqueName: \"kubernetes.io/projected/aea20d56-947f-4b93-9352-4a361fb40961-kube-api-access-7fcnz\") pod \"maas-api-8c9d8544c-t7vr6\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") " pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:22.695766 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:22.695671    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:23.081778 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:23.081752    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"]
Oct 30 21:49:23.084457 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:49:23.084429    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea20d56_947f_4b93_9352_4a361fb40961.slice/crio-a1a5d8ae98dd44fcbc24d43769e1c1416383b63a081968e224c11a867a0302af WatchSource:0}: Error finding container a1a5d8ae98dd44fcbc24d43769e1c1416383b63a081968e224c11a867a0302af: Status 404 returned error can't find the container with id a1a5d8ae98dd44fcbc24d43769e1c1416383b63a081968e224c11a867a0302af
Oct 30 21:49:23.122553 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:23.122523    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6" event={"ID":"aea20d56-947f-4b93-9352-4a361fb40961","Type":"ContainerStarted","Data":"a1a5d8ae98dd44fcbc24d43769e1c1416383b63a081968e224c11a867a0302af"}
Oct 30 21:49:28.199910 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:49:28.199878    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:49:29.149640 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:29.149594    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6" event={"ID":"aea20d56-947f-4b93-9352-4a361fb40961","Type":"ContainerStarted","Data":"6f44faf82d866bb8b03e098c8b53f40b0fed4ff6a7022da135107dffba72751c"}
Oct 30 21:49:29.149837 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:29.149706    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:29.266402 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:29.266287    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6" podStartSLOduration=1.410510768 podStartE2EDuration="7.26627366s" totalImagesPullingTime="5.855762892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:29.265270449 +0000 UTC m=+419.999633480" watchObservedRunningTime="2026-10-30 21:49:29.26627366 +0000 UTC m=+420.000636729"
Oct 30 21:49:35.159900 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:35.159870    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:36.177094 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.177058    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"]
Oct 30 21:49:36.180955 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.180936    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:36.267546 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.267514    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"]
Oct 30 21:49:36.273334 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.273305    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af1193f1-5a09-491e-ae7a-3b6ba045097a-metrics-certs\") pod \"maas-api-6c6fd44f6f-p7n4h\" (UID: \"af1193f1-5a09-491e-ae7a-3b6ba045097a\") " pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:36.273488 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.273361    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af1193f1-5a09-491e-ae7a-3b6ba045097a-maas-api-tls\") pod \"maas-api-6c6fd44f6f-p7n4h\" (UID: \"af1193f1-5a09-491e-ae7a-3b6ba045097a\") " pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:36.273488 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.273386    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af1193f1-5a09-491e-ae7a-3b6ba045097a-openshift-service-ca\") pod \"maas-api-6c6fd44f6f-p7n4h\" (UID: \"af1193f1-5a09-491e-ae7a-3b6ba045097a\") " pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:36.273597 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.273523    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnxqt\" (UniqueName: \"kubernetes.io/projected/af1193f1-5a09-491e-ae7a-3b6ba045097a-kube-api-access-jnxqt\") pod \"maas-api-6c6fd44f6f-p7n4h\" (UID: \"af1193f1-5a09-491e-ae7a-3b6ba045097a\") " pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:36.374723 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.374691    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnxqt\" (UniqueName: \"kubernetes.io/projected/af1193f1-5a09-491e-ae7a-3b6ba045097a-kube-api-access-jnxqt\") pod \"maas-api-6c6fd44f6f-p7n4h\" (UID: \"af1193f1-5a09-491e-ae7a-3b6ba045097a\") " pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:36.374927 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.374762    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af1193f1-5a09-491e-ae7a-3b6ba045097a-metrics-certs\") pod \"maas-api-6c6fd44f6f-p7n4h\" (UID: \"af1193f1-5a09-491e-ae7a-3b6ba045097a\") " pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:36.374927 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.374791    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af1193f1-5a09-491e-ae7a-3b6ba045097a-maas-api-tls\") pod \"maas-api-6c6fd44f6f-p7n4h\" (UID: \"af1193f1-5a09-491e-ae7a-3b6ba045097a\") " pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:36.374927 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.374806    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af1193f1-5a09-491e-ae7a-3b6ba045097a-openshift-service-ca\") pod \"maas-api-6c6fd44f6f-p7n4h\" (UID: \"af1193f1-5a09-491e-ae7a-3b6ba045097a\") " pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:36.375434 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.375412    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af1193f1-5a09-491e-ae7a-3b6ba045097a-openshift-service-ca\") pod \"maas-api-6c6fd44f6f-p7n4h\" (UID: \"af1193f1-5a09-491e-ae7a-3b6ba045097a\") " pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:36.377480 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.377459    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af1193f1-5a09-491e-ae7a-3b6ba045097a-metrics-certs\") pod \"maas-api-6c6fd44f6f-p7n4h\" (UID: \"af1193f1-5a09-491e-ae7a-3b6ba045097a\") " pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:36.377571 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.377501    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af1193f1-5a09-491e-ae7a-3b6ba045097a-maas-api-tls\") pod \"maas-api-6c6fd44f6f-p7n4h\" (UID: \"af1193f1-5a09-491e-ae7a-3b6ba045097a\") " pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:36.433878 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.433802    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnxqt\" (UniqueName: \"kubernetes.io/projected/af1193f1-5a09-491e-ae7a-3b6ba045097a-kube-api-access-jnxqt\") pod \"maas-api-6c6fd44f6f-p7n4h\" (UID: \"af1193f1-5a09-491e-ae7a-3b6ba045097a\") " pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:36.493365 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.493339    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:36.865279 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:36.865253    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"]
Oct 30 21:49:36.867714 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:49:36.867678    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1193f1_5a09_491e_ae7a_3b6ba045097a.slice/crio-9493b81c1d300fcfa3fc26bc71956fd971ce7e4150c5bfe74fe63ce26110a87c WatchSource:0}: Error finding container 9493b81c1d300fcfa3fc26bc71956fd971ce7e4150c5bfe74fe63ce26110a87c: Status 404 returned error can't find the container with id 9493b81c1d300fcfa3fc26bc71956fd971ce7e4150c5bfe74fe63ce26110a87c
Oct 30 21:49:37.184924 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:37.184836    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h" event={"ID":"af1193f1-5a09-491e-ae7a-3b6ba045097a","Type":"ContainerStarted","Data":"9493b81c1d300fcfa3fc26bc71956fd971ce7e4150c5bfe74fe63ce26110a87c"}
Oct 30 21:49:38.191189 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:38.191150    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h" event={"ID":"af1193f1-5a09-491e-ae7a-3b6ba045097a","Type":"ContainerStarted","Data":"c7d9051c0f0b7d6755466e7bb144d2908d46056dbcfb15cf23d5456d5dfe3f17"}
Oct 30 21:49:38.191583 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:38.191252    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:38.310566 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:38.310515    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h" podStartSLOduration=1.5413662540000002 podStartE2EDuration="2.310500413s" totalImagesPullingTime="769.134159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:38.310207458 +0000 UTC m=+429.044570515" watchObservedRunningTime="2026-10-30 21:49:38.310500413 +0000 UTC m=+429.044863444"
Oct 30 21:49:44.201043 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.201011    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c6fd44f6f-p7n4h"
Oct 30 21:49:44.369976 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.369940    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"]
Oct 30 21:49:44.370232 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.370162    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6" podUID="aea20d56-947f-4b93-9352-4a361fb40961" containerName="maas-api" containerID="cri-o://6f44faf82d866bb8b03e098c8b53f40b0fed4ff6a7022da135107dffba72751c" gracePeriod=30
Oct 30 21:49:44.623426 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.623398    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:44.751841 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.751769    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aea20d56-947f-4b93-9352-4a361fb40961-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aea20d56-947f-4b93-9352-4a361fb40961-maas-api-tls\") pod \"aea20d56-947f-4b93-9352-4a361fb40961\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") "
Oct 30 21:49:44.751841 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.751817    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aea20d56-947f-4b93-9352-4a361fb40961-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aea20d56-947f-4b93-9352-4a361fb40961-metrics-certs\") pod \"aea20d56-947f-4b93-9352-4a361fb40961\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") "
Oct 30 21:49:44.752070 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.751866    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea20d56-947f-4b93-9352-4a361fb40961-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aea20d56-947f-4b93-9352-4a361fb40961-openshift-service-ca\") pod \"aea20d56-947f-4b93-9352-4a361fb40961\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") "
Oct 30 21:49:44.752070 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.751888    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea20d56-947f-4b93-9352-4a361fb40961-kube-api-access-7fcnz\" (UniqueName: \"kubernetes.io/projected/aea20d56-947f-4b93-9352-4a361fb40961-kube-api-access-7fcnz\") pod \"aea20d56-947f-4b93-9352-4a361fb40961\" (UID: \"aea20d56-947f-4b93-9352-4a361fb40961\") "
Oct 30 21:49:44.752306 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.752275    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea20d56-947f-4b93-9352-4a361fb40961-openshift-service-ca" pod "aea20d56-947f-4b93-9352-4a361fb40961" (UID: "aea20d56-947f-4b93-9352-4a361fb40961"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:44.754428 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.754387    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea20d56-947f-4b93-9352-4a361fb40961-kube-api-access-7fcnz" pod "aea20d56-947f-4b93-9352-4a361fb40961" (UID: "aea20d56-947f-4b93-9352-4a361fb40961"). InnerVolumeSpecName "kube-api-access-7fcnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:44.754569 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.754540    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aea20d56-947f-4b93-9352-4a361fb40961-metrics-certs" pod "aea20d56-947f-4b93-9352-4a361fb40961" (UID: "aea20d56-947f-4b93-9352-4a361fb40961"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:49:44.755268 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.755242    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aea20d56-947f-4b93-9352-4a361fb40961-maas-api-tls" pod "aea20d56-947f-4b93-9352-4a361fb40961" (UID: "aea20d56-947f-4b93-9352-4a361fb40961"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:49:44.853335 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.853299    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aea20d56-947f-4b93-9352-4a361fb40961-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:44.853335 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.853327    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aea20d56-947f-4b93-9352-4a361fb40961-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:44.853335 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.853337    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aea20d56-947f-4b93-9352-4a361fb40961-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:44.853575 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:44.853348    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fcnz\" (UniqueName: \"kubernetes.io/projected/aea20d56-947f-4b93-9352-4a361fb40961-kube-api-access-7fcnz\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:45.221617 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:45.221585    4553 generic.go:356] "Generic (PLEG): container finished" podID="aea20d56-947f-4b93-9352-4a361fb40961" containerID="6f44faf82d866bb8b03e098c8b53f40b0fed4ff6a7022da135107dffba72751c" exitCode=0
Oct 30 21:49:45.222114 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:45.221630    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"
Oct 30 21:49:45.222114 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:45.221670    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6" event={"ID":"aea20d56-947f-4b93-9352-4a361fb40961","Type":"ContainerDied","Data":"6f44faf82d866bb8b03e098c8b53f40b0fed4ff6a7022da135107dffba72751c"}
Oct 30 21:49:45.222114 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:45.221733    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6" event={"ID":"aea20d56-947f-4b93-9352-4a361fb40961","Type":"ContainerDied","Data":"a1a5d8ae98dd44fcbc24d43769e1c1416383b63a081968e224c11a867a0302af"}
Oct 30 21:49:45.222114 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:45.221766    4553 scope.go:122] "RemoveContainer" containerID="6f44faf82d866bb8b03e098c8b53f40b0fed4ff6a7022da135107dffba72751c"
Oct 30 21:49:45.231035 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:45.231011    4553 scope.go:122] "RemoveContainer" containerID="6f44faf82d866bb8b03e098c8b53f40b0fed4ff6a7022da135107dffba72751c"
Oct 30 21:49:45.231386 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:49:45.231363    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f44faf82d866bb8b03e098c8b53f40b0fed4ff6a7022da135107dffba72751c\": container with ID starting with 6f44faf82d866bb8b03e098c8b53f40b0fed4ff6a7022da135107dffba72751c not found: ID does not exist" containerID="6f44faf82d866bb8b03e098c8b53f40b0fed4ff6a7022da135107dffba72751c"
Oct 30 21:49:45.231481 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:45.231399    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f44faf82d866bb8b03e098c8b53f40b0fed4ff6a7022da135107dffba72751c"} err="failed to get container status \"6f44faf82d866bb8b03e098c8b53f40b0fed4ff6a7022da135107dffba72751c\": rpc error: code = NotFound desc = could not find container \"6f44faf82d866bb8b03e098c8b53f40b0fed4ff6a7022da135107dffba72751c\": container with ID starting with 6f44faf82d866bb8b03e098c8b53f40b0fed4ff6a7022da135107dffba72751c not found: ID does not exist"
Oct 30 21:49:45.398259 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:45.398230    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"]
Oct 30 21:49:45.447015 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:45.446982    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8c9d8544c-t7vr6"]
Oct 30 21:49:47.391524 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:49:47.391491    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea20d56-947f-4b93-9352-4a361fb40961" path="/var/lib/kubelet/pods/aea20d56-947f-4b93-9352-4a361fb40961/volumes"
Oct 30 21:49:58.199646 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:49:58.199607    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:50:28.200091 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:50:28.200052    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:50:35.158494 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.158462    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-x6vsn"]
Oct 30 21:50:35.158987 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.158808    4553 cpu_manager.go:414] "containerMap: removing container" podUID="aea20d56-947f-4b93-9352-4a361fb40961" containerName="maas-api"
Oct 30 21:50:35.158987 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.158819    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea20d56-947f-4b93-9352-4a361fb40961" containerName="maas-api"
Oct 30 21:50:35.158987 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.158892    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea20d56-947f-4b93-9352-4a361fb40961" containerName="maas-api"
Oct 30 21:50:35.163222 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.163205    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn"
Oct 30 21:50:35.218765 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.218731    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-x6vsn"]
Oct 30 21:50:35.267631 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.267589    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 21:50:35.267631 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.267606    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 21:50:35.267847 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.267701    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-l2xfv\""
Oct 30 21:50:35.296359 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.296323    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6krv\" (UniqueName: \"kubernetes.io/projected/e4ba72ff-936c-4207-beff-86eb80e1cd90-kube-api-access-p6krv\") pod \"ai-gateway-controller-7d85675f98-x6vsn\" (UID: \"e4ba72ff-936c-4207-beff-86eb80e1cd90\") " pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn"
Oct 30 21:50:35.296539 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.296376    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ba72ff-936c-4207-beff-86eb80e1cd90-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-x6vsn\" (UID: \"e4ba72ff-936c-4207-beff-86eb80e1cd90\") " pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn"
Oct 30 21:50:35.397214 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.397181    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ba72ff-936c-4207-beff-86eb80e1cd90-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-x6vsn\" (UID: \"e4ba72ff-936c-4207-beff-86eb80e1cd90\") " pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn"
Oct 30 21:50:35.397380 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.397267    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6krv\" (UniqueName: \"kubernetes.io/projected/e4ba72ff-936c-4207-beff-86eb80e1cd90-kube-api-access-p6krv\") pod \"ai-gateway-controller-7d85675f98-x6vsn\" (UID: \"e4ba72ff-936c-4207-beff-86eb80e1cd90\") " pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn"
Oct 30 21:50:35.399776 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.399744    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ba72ff-936c-4207-beff-86eb80e1cd90-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-x6vsn\" (UID: \"e4ba72ff-936c-4207-beff-86eb80e1cd90\") " pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn"
Oct 30 21:50:35.455697 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.455624    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6krv\" (UniqueName: \"kubernetes.io/projected/e4ba72ff-936c-4207-beff-86eb80e1cd90-kube-api-access-p6krv\") pod \"ai-gateway-controller-7d85675f98-x6vsn\" (UID: \"e4ba72ff-936c-4207-beff-86eb80e1cd90\") " pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn"
Oct 30 21:50:35.475513 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.475483    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn"
Oct 30 21:50:35.846501 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:35.846471    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-x6vsn"]
Oct 30 21:50:35.848778 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:50:35.848749    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ba72ff_936c_4207_beff_86eb80e1cd90.slice/crio-2c778224e9452ace09085a810830af52b4f8e8882a8e15d30c27fee22d5a20da WatchSource:0}: Error finding container 2c778224e9452ace09085a810830af52b4f8e8882a8e15d30c27fee22d5a20da: Status 404 returned error can't find the container with id 2c778224e9452ace09085a810830af52b4f8e8882a8e15d30c27fee22d5a20da
Oct 30 21:50:36.430534 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:36.430498    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn" event={"ID":"e4ba72ff-936c-4207-beff-86eb80e1cd90","Type":"ContainerStarted","Data":"2c778224e9452ace09085a810830af52b4f8e8882a8e15d30c27fee22d5a20da"}
Oct 30 21:50:39.445983 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:39.445947    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn" event={"ID":"e4ba72ff-936c-4207-beff-86eb80e1cd90","Type":"ContainerStarted","Data":"5135193f120fedc522b91866bd73f322f6cb6311bfb3162eafcb45ffc0a737dc"}
Oct 30 21:50:39.446502 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:39.446054    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn"
Oct 30 21:50:39.564876 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:39.564824    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn" podStartSLOduration=1.944199986 podStartE2EDuration="4.564809961s" totalImagesPullingTime="2.620609975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:39.564452569 +0000 UTC m=+490.298815601" watchObservedRunningTime="2026-10-30 21:50:39.564809961 +0000 UTC m=+490.299172991"
Oct 30 21:50:42.796254 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:42.796214    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-2szkr"]
Oct 30 21:50:42.799667 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:42.799647    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2szkr"
Oct 30 21:50:42.861304 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:42.861263    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 21:50:42.911156 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:42.911099    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-58sz5\""
Oct 30 21:50:42.912613 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:42.912590    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-2szkr"]
Oct 30 21:50:42.964445 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:42.964411    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzmgs\" (UniqueName: \"kubernetes.io/projected/78f6d7aa-e3b7-45f5-b68a-d6935dda41e6-kube-api-access-kzmgs\") pod \"payload-pre-processing-7b98bb678c-2szkr\" (UID: \"78f6d7aa-e3b7-45f5-b68a-d6935dda41e6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2szkr"
Oct 30 21:50:42.964646 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:42.964492    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f6d7aa-e3b7-45f5-b68a-d6935dda41e6-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-2szkr\" (UID: \"78f6d7aa-e3b7-45f5-b68a-d6935dda41e6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2szkr"
Oct 30 21:50:42.964918 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:42.964897    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-hg9wk"]
Oct 30 21:50:42.968521 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:42.968495    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-hg9wk"
Oct 30 21:50:43.025780 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:43.025748    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-hg9wk"]
Oct 30 21:50:43.065144 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:43.065051    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d7db25-7800-4217-a2fb-4396ba266c73-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-hg9wk\" (UID: \"05d7db25-7800-4217-a2fb-4396ba266c73\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-hg9wk"
Oct 30 21:50:43.065144 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:43.065116    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzmgs\" (UniqueName: \"kubernetes.io/projected/78f6d7aa-e3b7-45f5-b68a-d6935dda41e6-kube-api-access-kzmgs\") pod \"payload-pre-processing-7b98bb678c-2szkr\" (UID: \"78f6d7aa-e3b7-45f5-b68a-d6935dda41e6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2szkr"
Oct 30 21:50:43.065340 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:43.065207    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-642v9\" (UniqueName: \"kubernetes.io/projected/05d7db25-7800-4217-a2fb-4396ba266c73-kube-api-access-642v9\") pod \"payload-processing-55fbbbf9dc-hg9wk\" (UID: \"05d7db25-7800-4217-a2fb-4396ba266c73\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-hg9wk"
Oct 30 21:50:43.065340 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:43.065262    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f6d7aa-e3b7-45f5-b68a-d6935dda41e6-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-2szkr\" (UID: \"78f6d7aa-e3b7-45f5-b68a-d6935dda41e6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2szkr"
Oct 30 21:50:43.065970 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:43.065949    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f6d7aa-e3b7-45f5-b68a-d6935dda41e6-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-2szkr\" (UID: \"78f6d7aa-e3b7-45f5-b68a-d6935dda41e6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2szkr"
Oct 30 21:50:43.123972 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:43.123936    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzmgs\" (UniqueName: \"kubernetes.io/projected/78f6d7aa-e3b7-45f5-b68a-d6935dda41e6-kube-api-access-kzmgs\") pod \"payload-pre-processing-7b98bb678c-2szkr\" (UID: \"78f6d7aa-e3b7-45f5-b68a-d6935dda41e6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-2szkr"
Oct 30 21:50:43.165993 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:43.165959    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-642v9\" (UniqueName: \"kubernetes.io/projected/05d7db25-7800-4217-a2fb-4396ba266c73-kube-api-access-642v9\") pod \"payload-processing-55fbbbf9dc-hg9wk\" (UID: \"05d7db25-7800-4217-a2fb-4396ba266c73\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-hg9wk"
Oct 30 21:50:43.166224 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:43.166025    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d7db25-7800-4217-a2fb-4396ba266c73-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-hg9wk\" (UID: \"05d7db25-7800-4217-a2fb-4396ba266c73\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-hg9wk"
Oct 30 21:50:43.166704 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:43.166684    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05d7db25-7800-4217-a2fb-4396ba266c73-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-hg9wk\" (UID: \"05d7db25-7800-4217-a2fb-4396ba266c73\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-hg9wk"
Oct 30 21:50:43.223523 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:43.223494    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-642v9\" (UniqueName: \"kubernetes.io/projected/05d7db25-7800-4217-a2fb-4396ba266c73-kube-api-access-642v9\") pod \"payload-processing-55fbbbf9dc-hg9wk\" (UID: \"05d7db25-7800-4217-a2fb-4396ba266c73\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-hg9wk"
Oct 30 21:50:43.280799 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:43.280759    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-hg9wk"
Oct 30 21:50:43.413394 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:43.413364    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2szkr"
Oct 30 21:50:43.649028 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:43.649002    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-hg9wk"]
Oct 30 21:50:43.651329 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:50:43.651302    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d7db25_7800_4217_a2fb_4396ba266c73.slice/crio-6a19ee46b5137c8f75e18a416496346496c90db9d05c3332e5b028f10e10b32f WatchSource:0}: Error finding container 6a19ee46b5137c8f75e18a416496346496c90db9d05c3332e5b028f10e10b32f: Status 404 returned error can't find the container with id 6a19ee46b5137c8f75e18a416496346496c90db9d05c3332e5b028f10e10b32f
Oct 30 21:50:43.789090 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:43.789058    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-2szkr"]
Oct 30 21:50:43.791429 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:50:43.791400    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f6d7aa_e3b7_45f5_b68a_d6935dda41e6.slice/crio-7bcf279727e48ceef9ef135a5da354ce6ef46d03bb7022c4402b26fa31a6aef9 WatchSource:0}: Error finding container 7bcf279727e48ceef9ef135a5da354ce6ef46d03bb7022c4402b26fa31a6aef9: Status 404 returned error can't find the container with id 7bcf279727e48ceef9ef135a5da354ce6ef46d03bb7022c4402b26fa31a6aef9
Oct 30 21:50:44.469898 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:44.469863    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-hg9wk" event={"ID":"05d7db25-7800-4217-a2fb-4396ba266c73","Type":"ContainerStarted","Data":"6a19ee46b5137c8f75e18a416496346496c90db9d05c3332e5b028f10e10b32f"}
Oct 30 21:50:44.470985 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:44.470963    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2szkr" event={"ID":"78f6d7aa-e3b7-45f5-b68a-d6935dda41e6","Type":"ContainerStarted","Data":"7bcf279727e48ceef9ef135a5da354ce6ef46d03bb7022c4402b26fa31a6aef9"}
Oct 30 21:50:47.486215 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:47.486181    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-hg9wk" event={"ID":"05d7db25-7800-4217-a2fb-4396ba266c73","Type":"ContainerStarted","Data":"0d33470604740c5a269d7cb722fd4cf5db678e2e5e6fcde1cdb7998c1cb054e7"}
Oct 30 21:50:47.486689 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:47.486301    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-hg9wk"
Oct 30 21:50:47.487691 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:47.487665    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2szkr" event={"ID":"78f6d7aa-e3b7-45f5-b68a-d6935dda41e6","Type":"ContainerStarted","Data":"f8bcfb22b74090132493554e7abcaa156cdd117f0bd20afc9288d34b36900d04"}
Oct 30 21:50:47.487813 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:47.487800    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2szkr"
Oct 30 21:50:47.602018 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:47.601961    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-hg9wk" podStartSLOduration=2.504700003 podStartE2EDuration="5.601947168s" totalImagesPullingTime="3.097247165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:47.601109482 +0000 UTC m=+498.335472515" watchObservedRunningTime="2026-10-30 21:50:47.601947168 +0000 UTC m=+498.336310198"
Oct 30 21:50:47.717321 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:47.717263    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2szkr" podStartSLOduration=2.846339923 podStartE2EDuration="5.717247694s" totalImagesPullingTime="2.870907771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:47.716198637 +0000 UTC m=+498.450561669" watchObservedRunningTime="2026-10-30 21:50:47.717247694 +0000 UTC m=+498.451610721"
Oct 30 21:50:50.452965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:50.452934    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn"
Oct 30 21:50:50.781028 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:50.780953    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg"]
Oct 30 21:50:50.784772 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:50.784753    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg"
Oct 30 21:50:50.842746 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:50.842717    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg"]
Oct 30 21:50:50.937368 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:50.937334    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf6aad67-bdb9-4fae-ab13-cf8445563b6c-metrics-certs\") pod \"ai-gateway-controller-6467c7bcbb-q6mdg\" (UID: \"cf6aad67-bdb9-4fae-ab13-cf8445563b6c\") " pod="opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg"
Oct 30 21:50:50.937565 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:50.937414    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nzh5\" (UniqueName: \"kubernetes.io/projected/cf6aad67-bdb9-4fae-ab13-cf8445563b6c-kube-api-access-7nzh5\") pod \"ai-gateway-controller-6467c7bcbb-q6mdg\" (UID: \"cf6aad67-bdb9-4fae-ab13-cf8445563b6c\") " pod="opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg"
Oct 30 21:50:51.043236 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:51.039319    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf6aad67-bdb9-4fae-ab13-cf8445563b6c-metrics-certs\") pod \"ai-gateway-controller-6467c7bcbb-q6mdg\" (UID: \"cf6aad67-bdb9-4fae-ab13-cf8445563b6c\") " pod="opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg"
Oct 30 21:50:51.043236 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:51.039457    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzh5\" (UniqueName: \"kubernetes.io/projected/cf6aad67-bdb9-4fae-ab13-cf8445563b6c-kube-api-access-7nzh5\") pod \"ai-gateway-controller-6467c7bcbb-q6mdg\" (UID: \"cf6aad67-bdb9-4fae-ab13-cf8445563b6c\") " pod="opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg"
Oct 30 21:50:51.047729 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:51.047689    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf6aad67-bdb9-4fae-ab13-cf8445563b6c-metrics-certs\") pod \"ai-gateway-controller-6467c7bcbb-q6mdg\" (UID: \"cf6aad67-bdb9-4fae-ab13-cf8445563b6c\") " pod="opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg"
Oct 30 21:50:51.097337 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:51.097302    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nzh5\" (UniqueName: \"kubernetes.io/projected/cf6aad67-bdb9-4fae-ab13-cf8445563b6c-kube-api-access-7nzh5\") pod \"ai-gateway-controller-6467c7bcbb-q6mdg\" (UID: \"cf6aad67-bdb9-4fae-ab13-cf8445563b6c\") " pod="opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg"
Oct 30 21:50:51.397843 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:51.397806    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg"
Oct 30 21:50:51.766832 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:51.766809    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg"]
Oct 30 21:50:51.769483 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:50:51.769451    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6aad67_bdb9_4fae_ab13_cf8445563b6c.slice/crio-0fbdd48a4304a1586fba39d07a1c472f7a740ce38c975647f674410aea24af18 WatchSource:0}: Error finding container 0fbdd48a4304a1586fba39d07a1c472f7a740ce38c975647f674410aea24af18: Status 404 returned error can't find the container with id 0fbdd48a4304a1586fba39d07a1c472f7a740ce38c975647f674410aea24af18
Oct 30 21:50:52.512260 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:52.512227    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg" event={"ID":"cf6aad67-bdb9-4fae-ab13-cf8445563b6c","Type":"ContainerStarted","Data":"0fbdd48a4304a1586fba39d07a1c472f7a740ce38c975647f674410aea24af18"}
Oct 30 21:50:53.517873 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:53.517840    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg" event={"ID":"cf6aad67-bdb9-4fae-ab13-cf8445563b6c","Type":"ContainerStarted","Data":"db00441890d887809ac1aaa850ca71b69013db0325fd1b0eef4462b74221013a"}
Oct 30 21:50:53.518404 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:53.517982    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg"
Oct 30 21:50:53.633911 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:53.633859    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg" podStartSLOduration=2.787350816 podStartE2EDuration="3.633842486s" totalImagesPullingTime="846.49167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:53.632716977 +0000 UTC m=+504.367080034" watchObservedRunningTime="2026-10-30 21:50:53.633842486 +0000 UTC m=+504.368205514"
Oct 30 21:50:58.199316 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:50:58.199272    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:50:58.494690 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:58.494617    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-hg9wk"
Oct 30 21:50:58.494690 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:50:58.494690    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-2szkr"
Oct 30 21:51:04.524617 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:04.524587    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6467c7bcbb-q6mdg"
Oct 30 21:51:04.692591 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:04.692559    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-x6vsn"]
Oct 30 21:51:04.692816 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:04.692780    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn" podUID="e4ba72ff-936c-4207-beff-86eb80e1cd90" containerName="manager" containerID="cri-o://5135193f120fedc522b91866bd73f322f6cb6311bfb3162eafcb45ffc0a737dc" gracePeriod=30
Oct 30 21:51:04.937454 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:04.937430    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn"
Oct 30 21:51:04.951302 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:04.951281    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ba72ff-936c-4207-beff-86eb80e1cd90-kube-api-access-p6krv\" (UniqueName: \"kubernetes.io/projected/e4ba72ff-936c-4207-beff-86eb80e1cd90-kube-api-access-p6krv\") pod \"e4ba72ff-936c-4207-beff-86eb80e1cd90\" (UID: \"e4ba72ff-936c-4207-beff-86eb80e1cd90\") "
Oct 30 21:51:04.951433 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:04.951350    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4ba72ff-936c-4207-beff-86eb80e1cd90-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ba72ff-936c-4207-beff-86eb80e1cd90-metrics-certs\") pod \"e4ba72ff-936c-4207-beff-86eb80e1cd90\" (UID: \"e4ba72ff-936c-4207-beff-86eb80e1cd90\") "
Oct 30 21:51:04.953532 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:04.953503    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4ba72ff-936c-4207-beff-86eb80e1cd90-metrics-certs" pod "e4ba72ff-936c-4207-beff-86eb80e1cd90" (UID: "e4ba72ff-936c-4207-beff-86eb80e1cd90"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:51:04.953648 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:04.953555    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ba72ff-936c-4207-beff-86eb80e1cd90-kube-api-access-p6krv" pod "e4ba72ff-936c-4207-beff-86eb80e1cd90" (UID: "e4ba72ff-936c-4207-beff-86eb80e1cd90"). InnerVolumeSpecName "kube-api-access-p6krv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:05.052733 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:05.052698    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6krv\" (UniqueName: \"kubernetes.io/projected/e4ba72ff-936c-4207-beff-86eb80e1cd90-kube-api-access-p6krv\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:05.052733 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:05.052728    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4ba72ff-936c-4207-beff-86eb80e1cd90-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:05.575559 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:05.575526    4553 generic.go:356] "Generic (PLEG): container finished" podID="e4ba72ff-936c-4207-beff-86eb80e1cd90" containerID="5135193f120fedc522b91866bd73f322f6cb6311bfb3162eafcb45ffc0a737dc" exitCode=0
Oct 30 21:51:05.576070 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:05.575574    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn"
Oct 30 21:51:05.576070 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:05.575614    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn" event={"ID":"e4ba72ff-936c-4207-beff-86eb80e1cd90","Type":"ContainerDied","Data":"5135193f120fedc522b91866bd73f322f6cb6311bfb3162eafcb45ffc0a737dc"}
Oct 30 21:51:05.576070 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:05.575681    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-x6vsn" event={"ID":"e4ba72ff-936c-4207-beff-86eb80e1cd90","Type":"ContainerDied","Data":"2c778224e9452ace09085a810830af52b4f8e8882a8e15d30c27fee22d5a20da"}
Oct 30 21:51:05.576070 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:05.575712    4553 scope.go:122] "RemoveContainer" containerID="5135193f120fedc522b91866bd73f322f6cb6311bfb3162eafcb45ffc0a737dc"
Oct 30 21:51:05.584840 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:05.584819    4553 scope.go:122] "RemoveContainer" containerID="5135193f120fedc522b91866bd73f322f6cb6311bfb3162eafcb45ffc0a737dc"
Oct 30 21:51:05.585204 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:51:05.585184    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5135193f120fedc522b91866bd73f322f6cb6311bfb3162eafcb45ffc0a737dc\": container with ID starting with 5135193f120fedc522b91866bd73f322f6cb6311bfb3162eafcb45ffc0a737dc not found: ID does not exist" containerID="5135193f120fedc522b91866bd73f322f6cb6311bfb3162eafcb45ffc0a737dc"
Oct 30 21:51:05.585287 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:05.585221    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5135193f120fedc522b91866bd73f322f6cb6311bfb3162eafcb45ffc0a737dc"} err="failed to get container status \"5135193f120fedc522b91866bd73f322f6cb6311bfb3162eafcb45ffc0a737dc\": rpc error: code = NotFound desc = could not find container \"5135193f120fedc522b91866bd73f322f6cb6311bfb3162eafcb45ffc0a737dc\": container with ID starting with 5135193f120fedc522b91866bd73f322f6cb6311bfb3162eafcb45ffc0a737dc not found: ID does not exist"
Oct 30 21:51:05.741691 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:05.741605    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-x6vsn"]
Oct 30 21:51:05.791985 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:05.791956    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-x6vsn"]
Oct 30 21:51:07.391111 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:07.391075    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ba72ff-936c-4207-beff-86eb80e1cd90" path="/var/lib/kubelet/pods/e4ba72ff-936c-4207-beff-86eb80e1cd90/volumes"
Oct 30 21:51:24.675267 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.675220    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"]
Oct 30 21:51:24.675776 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.675604    4553 cpu_manager.go:414] "containerMap: removing container" podUID="e4ba72ff-936c-4207-beff-86eb80e1cd90" containerName="manager"
Oct 30 21:51:24.675776 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.675615    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ba72ff-936c-4207-beff-86eb80e1cd90" containerName="manager"
Oct 30 21:51:24.675776 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.675694    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ba72ff-936c-4207-beff-86eb80e1cd90" containerName="manager"
Oct 30 21:51:24.678676 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.678661    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.731924 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.731894    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 21:51:24.782114 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.782081    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 21:51:24.782114 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.782105    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dbz24\""
Oct 30 21:51:24.782349 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.782327    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 21:51:24.783736 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.783720    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"]
Oct 30 21:51:24.835665 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.835632    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.835856 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.835713    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.835856 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.835732    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.835856 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.835761    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.835856 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.835785    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.836019 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.835971    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdt7g\" (UniqueName: \"kubernetes.io/projected/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-kube-api-access-fdt7g\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.936991 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.936903    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.936991 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.936947    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.937273 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.936993    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdt7g\" (UniqueName: \"kubernetes.io/projected/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-kube-api-access-fdt7g\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.937273 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.937022    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.937273 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.937045    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.937273 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.937060    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.937486 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.937469    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.937528 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.937488    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.937581 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.937556    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.939441 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.939418    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.939700 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.939682    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:24.998779 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:24.998746    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdt7g\" (UniqueName: \"kubernetes.io/projected/8526d260-3d4c-42e8-9bb6-194d1b4fb53d-kube-api-access-fdt7g\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht\" (UID: \"8526d260-3d4c-42e8-9bb6-194d1b4fb53d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:25.291671 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:25.291586    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:25.666652 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:25.666628    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"]
Oct 30 21:51:25.669098 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:51:25.669062    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8526d260_3d4c_42e8_9bb6_194d1b4fb53d.slice/crio-c4fb6bef4fc42aba26338288122d7f3f139d26cfc9bcf2f3c870653fdfba9fa0 WatchSource:0}: Error finding container c4fb6bef4fc42aba26338288122d7f3f139d26cfc9bcf2f3c870653fdfba9fa0: Status 404 returned error can't find the container with id c4fb6bef4fc42aba26338288122d7f3f139d26cfc9bcf2f3c870653fdfba9fa0
Oct 30 21:51:26.677185 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:26.677146    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht" event={"ID":"8526d260-3d4c-42e8-9bb6-194d1b4fb53d","Type":"ContainerStarted","Data":"c4fb6bef4fc42aba26338288122d7f3f139d26cfc9bcf2f3c870653fdfba9fa0"}
Oct 30 21:51:28.202693 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:51:28.202648    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:51:28.687964 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:28.687923    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht" event={"ID":"8526d260-3d4c-42e8-9bb6-194d1b4fb53d","Type":"ContainerStarted","Data":"66e5dfaece35fb4d98d1ab3cb4ddf783d2df94db366b0ffc4d454441907ed073"}
Oct 30 21:51:28.807361 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:28.807306    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht" podStartSLOduration=2.232940568 podStartE2EDuration="4.807289731s" totalImagesPullingTime="2.574349163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:28.805086046 +0000 UTC m=+539.539449080" watchObservedRunningTime="2026-10-30 21:51:28.807289731 +0000 UTC m=+539.541652762"
Oct 30 21:51:29.291929 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:29.291899    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:32.323944 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.323906    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"]
Oct 30 21:51:32.328756 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.328729    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.387640 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.387605    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 21:51:32.389649 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.389618    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"]
Oct 30 21:51:32.507451 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.507417    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.507636 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.507460    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.507636 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.507501    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.507636 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.507533    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.507636 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.507547    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.507636 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.507564    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-984bp\" (UniqueName: \"kubernetes.io/projected/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-kube-api-access-984bp\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.608804 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.608764    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.608804 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.608811    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.609028 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.608847    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.609028 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.608872    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.609028 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.608886    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.609028 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.608911    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-984bp\" (UniqueName: \"kubernetes.io/projected/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-kube-api-access-984bp\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.609337 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.609314    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.609406 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.609349    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.609406 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.609321    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.611375 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.611356    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.611516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.611499    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.667711 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.667672    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-984bp\" (UniqueName: \"kubernetes.io/projected/deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8-kube-api-access-984bp\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5xgvw\" (UID: \"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:32.941960 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:32.941882    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:33.318600 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:33.318575    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"]
Oct 30 21:51:33.320926 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:51:33.320898    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeeba91c_c79c_4d40_aea6_ae9e8cc3f4c8.slice/crio-bd9167a1e047744b4349b68b042661a0e4003a03ab90217267efc91970728194 WatchSource:0}: Error finding container bd9167a1e047744b4349b68b042661a0e4003a03ab90217267efc91970728194: Status 404 returned error can't find the container with id bd9167a1e047744b4349b68b042661a0e4003a03ab90217267efc91970728194
Oct 30 21:51:33.717789 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:33.717757    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw" event={"ID":"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8","Type":"ContainerStarted","Data":"bd9167a1e047744b4349b68b042661a0e4003a03ab90217267efc91970728194"}
Oct 30 21:51:34.724420 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:34.724385    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw" event={"ID":"deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8","Type":"ContainerStarted","Data":"1e2d31e6090ee8a70f2d41e2a5135752c0cc8c9e3924efcdc9dc49fdcca2d705"}
Oct 30 21:51:34.849903 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:34.849849    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw" podStartSLOduration=2.263872493 podStartE2EDuration="2.849833902s" totalImagesPullingTime="585.961409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:34.847931531 +0000 UTC m=+545.582294561" watchObservedRunningTime="2026-10-30 21:51:34.849833902 +0000 UTC m=+545.584196976"
Oct 30 21:51:34.942857 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:34.942824    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:35.292745 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:35.292701    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:35.305392 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:35.305366    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:35.739983 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:35.739949    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht"
Oct 30 21:51:42.343313 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.343277    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"]
Oct 30 21:51:42.346634 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.346616    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.403256 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.403209    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9867d053-6ea1-4062-814a-afdbaa4993f9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.403256 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.403253    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqg9b\" (UniqueName: \"kubernetes.io/projected/9867d053-6ea1-4062-814a-afdbaa4993f9-kube-api-access-xqg9b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.403524 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.403356    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9867d053-6ea1-4062-814a-afdbaa4993f9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.403524 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.403449    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9867d053-6ea1-4062-814a-afdbaa4993f9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.403524 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.403479    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9867d053-6ea1-4062-814a-afdbaa4993f9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.403524 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.403518    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9867d053-6ea1-4062-814a-afdbaa4993f9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.449818 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.449792    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 21:51:42.451375 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.451355    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"]
Oct 30 21:51:42.504004 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.503973    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9867d053-6ea1-4062-814a-afdbaa4993f9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.504242 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.504039    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9867d053-6ea1-4062-814a-afdbaa4993f9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.504242 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.504068    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9867d053-6ea1-4062-814a-afdbaa4993f9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.504242 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.504113    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9867d053-6ea1-4062-814a-afdbaa4993f9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.504242 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.504203    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9867d053-6ea1-4062-814a-afdbaa4993f9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.504242 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.504230    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqg9b\" (UniqueName: \"kubernetes.io/projected/9867d053-6ea1-4062-814a-afdbaa4993f9-kube-api-access-xqg9b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.504540 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.504516    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9867d053-6ea1-4062-814a-afdbaa4993f9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.504603 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.504580    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9867d053-6ea1-4062-814a-afdbaa4993f9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.504705 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.504685    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9867d053-6ea1-4062-814a-afdbaa4993f9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.506527 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.506503    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9867d053-6ea1-4062-814a-afdbaa4993f9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.506764 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.506748    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9867d053-6ea1-4062-814a-afdbaa4993f9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.565542 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.565509    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqg9b\" (UniqueName: \"kubernetes.io/projected/9867d053-6ea1-4062-814a-afdbaa4993f9-kube-api-access-xqg9b\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq\" (UID: \"9867d053-6ea1-4062-814a-afdbaa4993f9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.660039 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.659946    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:42.942608 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.942529    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:42.955170 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:42.955118    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:43.038130 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:43.038105    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"]
Oct 30 21:51:43.040247 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:51:43.040217    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9867d053_6ea1_4062_814a_afdbaa4993f9.slice/crio-f918598929db88787fe2b99882aa0dea0791dcb25e0b9771e2dd022c3dd03a80 WatchSource:0}: Error finding container f918598929db88787fe2b99882aa0dea0791dcb25e0b9771e2dd022c3dd03a80: Status 404 returned error can't find the container with id f918598929db88787fe2b99882aa0dea0791dcb25e0b9771e2dd022c3dd03a80
Oct 30 21:51:43.764893 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:43.764853    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq" event={"ID":"9867d053-6ea1-4062-814a-afdbaa4993f9","Type":"ContainerStarted","Data":"9fcdacce25fa490ac244bf91dcb767a134d697a2dc9eed2d2f96447b8db4ca75"}
Oct 30 21:51:43.765364 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:43.764926    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq" event={"ID":"9867d053-6ea1-4062-814a-afdbaa4993f9","Type":"ContainerStarted","Data":"f918598929db88787fe2b99882aa0dea0791dcb25e0b9771e2dd022c3dd03a80"}
Oct 30 21:51:43.776569 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:43.776543    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5xgvw"
Oct 30 21:51:43.883479 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:43.883375    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq" podStartSLOduration=1.307001509 podStartE2EDuration="1.883361924s" totalImagesPullingTime="576.360415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:43.882999176 +0000 UTC m=+554.617362207" watchObservedRunningTime="2026-10-30 21:51:43.883361924 +0000 UTC m=+554.617724954"
Oct 30 21:51:44.660565 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:44.660534    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:46.088300 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.088267    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bfhs7"]
Oct 30 21:51:46.091997 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.091978    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.133125 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.133088    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4978a88d-cbce-4aa8-92bc-9f2d8e273651-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.133318 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.133166    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4978a88d-cbce-4aa8-92bc-9f2d8e273651-home\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.133318 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.133256    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4978a88d-cbce-4aa8-92bc-9f2d8e273651-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.133318 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.133296    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpmp9\" (UniqueName: \"kubernetes.io/projected/4978a88d-cbce-4aa8-92bc-9f2d8e273651-kube-api-access-jpmp9\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.133446 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.133377    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4978a88d-cbce-4aa8-92bc-9f2d8e273651-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.133446 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.133430    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4978a88d-cbce-4aa8-92bc-9f2d8e273651-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.195552 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.195519    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 21:51:46.197639 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.197611    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bfhs7"]
Oct 30 21:51:46.234207 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.234175    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4978a88d-cbce-4aa8-92bc-9f2d8e273651-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.234207 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.234218    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4978a88d-cbce-4aa8-92bc-9f2d8e273651-home\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.234483 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.234350    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4978a88d-cbce-4aa8-92bc-9f2d8e273651-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.234483 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.234390    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpmp9\" (UniqueName: \"kubernetes.io/projected/4978a88d-cbce-4aa8-92bc-9f2d8e273651-kube-api-access-jpmp9\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.234483 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.234424    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4978a88d-cbce-4aa8-92bc-9f2d8e273651-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.234652 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.234608    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4978a88d-cbce-4aa8-92bc-9f2d8e273651-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.234875 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.234846    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4978a88d-cbce-4aa8-92bc-9f2d8e273651-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.234995 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.234899    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4978a88d-cbce-4aa8-92bc-9f2d8e273651-home\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.234995 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.234953    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4978a88d-cbce-4aa8-92bc-9f2d8e273651-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.237228 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.237155    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4978a88d-cbce-4aa8-92bc-9f2d8e273651-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.238144 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.238080    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4978a88d-cbce-4aa8-92bc-9f2d8e273651-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.293203 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.293156    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpmp9\" (UniqueName: \"kubernetes.io/projected/4978a88d-cbce-4aa8-92bc-9f2d8e273651-kube-api-access-jpmp9\") pod \"echo-simulator-kserve-64b6f8f96c-bfhs7\" (UID: \"4978a88d-cbce-4aa8-92bc-9f2d8e273651\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.405811 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.405718    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:46.783715 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:46.783676    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bfhs7"]
Oct 30 21:51:46.785963 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:51:46.785934    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4978a88d_cbce_4aa8_92bc_9f2d8e273651.slice/crio-684246f9102d7e01b27ff4e3a2f08602bb2e20d6dad45eea76b944e4e507b1a5 WatchSource:0}: Error finding container 684246f9102d7e01b27ff4e3a2f08602bb2e20d6dad45eea76b944e4e507b1a5: Status 404 returned error can't find the container with id 684246f9102d7e01b27ff4e3a2f08602bb2e20d6dad45eea76b944e4e507b1a5
Oct 30 21:51:47.787853 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:47.787818    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7" event={"ID":"4978a88d-cbce-4aa8-92bc-9f2d8e273651","Type":"ContainerStarted","Data":"684246f9102d7e01b27ff4e3a2f08602bb2e20d6dad45eea76b944e4e507b1a5"}
Oct 30 21:51:49.798888 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:49.798858    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7" event={"ID":"4978a88d-cbce-4aa8-92bc-9f2d8e273651","Type":"ContainerStarted","Data":"ab7f548058cff2cbc187975c14acb0fa438c67ae6acf140d94a0093d62c3e395"}
Oct 30 21:51:49.920203 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:49.920150    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7" podStartSLOduration=1.649300277 podStartE2EDuration="3.920114598s" totalImagesPullingTime="2.270814321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:49.918655902 +0000 UTC m=+560.653018934" watchObservedRunningTime="2026-10-30 21:51:49.920114598 +0000 UTC m=+560.654477679"
Oct 30 21:51:50.406639 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:50.406604    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:52.660746 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:52.660707    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:52.673730 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:52.673701    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:52.830600 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:52.830576    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq"
Oct 30 21:51:56.406498 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:56.406457    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:56.419485 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:56.419453    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:56.850552 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:51:56.850521    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-bfhs7"
Oct 30 21:51:58.200326 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:51:58.200289    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:52:09.430216 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.430124    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"]
Oct 30 21:52:09.434501 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.434480    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.490647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.490614    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dz4rf\""
Oct 30 21:52:09.543706 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.543672    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"]
Oct 30 21:52:09.547421 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.547399    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.547505 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.547439    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.547505 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.547461    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.547609 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.547542    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsx87\" (UniqueName: \"kubernetes.io/projected/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-kube-api-access-jsx87\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.547652 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.547630    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.547724 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.547711    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.547758 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.547747    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.547796 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.547763    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.547851 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.547817    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.648814 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.648783    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.649012 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.648833    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.649012 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.648859    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.649012 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.648981    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.649233 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.649036    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsx87\" (UniqueName: \"kubernetes.io/projected/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-kube-api-access-jsx87\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.649233 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.649068    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.649233 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.649118    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.649233 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.649202    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.649233 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.649228    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.649465 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.649244    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.649465 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.649304    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.649465 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.649343    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.649595 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.649537    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.649987 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.649936    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.651918 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.651890    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.652535 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.652495    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.707648 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.707576    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsx87\" (UniqueName: \"kubernetes.io/projected/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-kube-api-access-jsx87\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.707648 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.707580    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-j4c2z\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:09.747764 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:09.747734    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:10.171514 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:10.171492    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"]
Oct 30 21:52:10.174047 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:52:10.174021    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b6e606_c225_4dd2_94c0_e8bfe42ee109.slice/crio-c8782dae5fa3f5fbd3bf752bc3a6f8eb295e46f3b5f6b0880d983d241f4dd674 WatchSource:0}: Error finding container c8782dae5fa3f5fbd3bf752bc3a6f8eb295e46f3b5f6b0880d983d241f4dd674: Status 404 returned error can't find the container with id c8782dae5fa3f5fbd3bf752bc3a6f8eb295e46f3b5f6b0880d983d241f4dd674
Oct 30 21:52:10.902287 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:10.902252    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z" event={"ID":"c2b6e606-c225-4dd2-94c0-e8bfe42ee109","Type":"ContainerStarted","Data":"c8782dae5fa3f5fbd3bf752bc3a6f8eb295e46f3b5f6b0880d983d241f4dd674"}
Oct 30 21:52:13.957837 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:13.957806    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:52:13.958151 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:13.957878    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:52:13.958151 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:13.957908    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:52:14.938158 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:14.938114    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z" event={"ID":"c2b6e606-c225-4dd2-94c0-e8bfe42ee109","Type":"ContainerStarted","Data":"8a10b957163558bea4ea4fb9711fcbd115b449a585cfc38f9b5ad1fd658e2d27"}
Oct 30 21:52:15.747979 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:15.747949    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:15.749519 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:15.749493    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-j4c2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:52:15.749651 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:15.749552    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z" podUID="c2b6e606-c225-4dd2-94c0-e8bfe42ee109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 21:52:16.748591 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:16.748555    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-j4c2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:52:16.748997 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:16.748608    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z" podUID="c2b6e606-c225-4dd2-94c0-e8bfe42ee109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 21:52:17.748120 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:17.748085    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-j4c2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:52:17.748328 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:17.748161    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z" podUID="c2b6e606-c225-4dd2-94c0-e8bfe42ee109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 21:52:18.748586 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:18.748555    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-j4c2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:52:18.748967 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:18.748606    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z" podUID="c2b6e606-c225-4dd2-94c0-e8bfe42ee109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 21:52:19.749079 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:19.749042    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-j4c2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:52:19.749521 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:19.749116    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z" podUID="c2b6e606-c225-4dd2-94c0-e8bfe42ee109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 21:52:20.748396 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:20.748349    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-j4c2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 21:52:20.748585 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:20.748453    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z" podUID="c2b6e606-c225-4dd2-94c0-e8bfe42ee109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 21:52:22.383070 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:22.383037    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:22.383624 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:22.383391    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:22.384511 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:22.384494    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 21:52:22.578008 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:22.577949    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z" podStartSLOduration=9.797116637 podStartE2EDuration="13.577934443s" totalImagesPullingTime="3.780817806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:15.157570125 +0000 UTC m=+585.891933156" watchObservedRunningTime="2026-10-30 21:52:22.577934443 +0000 UTC m=+593.312297530"
Oct 30 21:52:24.559399 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.559359    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"]
Oct 30 21:52:24.568459 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.568430    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.607630 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.607567    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20775493-c504-4316-9d6f-816952c83e9a-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.607630 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.607597    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflch\" (UniqueName: \"kubernetes.io/projected/20775493-c504-4316-9d6f-816952c83e9a-kube-api-access-rflch\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.607630 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.607619    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.607852 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.607689    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20775493-c504-4316-9d6f-816952c83e9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.607852 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.607749    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.607852 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.607793    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20775493-c504-4316-9d6f-816952c83e9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.607852 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.607847    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.607975 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.607864    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.607975 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.607911    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.627033 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.626997    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"]
Oct 30 21:52:24.708688 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.708655    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20775493-c504-4316-9d6f-816952c83e9a-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.708688 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.708693    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflch\" (UniqueName: \"kubernetes.io/projected/20775493-c504-4316-9d6f-816952c83e9a-kube-api-access-rflch\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.708940 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.708714    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.708940 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.708748    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20775493-c504-4316-9d6f-816952c83e9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.708940 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.708767    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.708940 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.708820    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20775493-c504-4316-9d6f-816952c83e9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.708940 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.708861    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.708940 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.708886    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.708940 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.708910    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.709269 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.709239    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.709329 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.709308    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.709329 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.709323    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.709449 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.709378    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.709548 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.709530    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20775493-c504-4316-9d6f-816952c83e9a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.711367 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.711341    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.711495 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.711477    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20775493-c504-4316-9d6f-816952c83e9a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.767306 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.767264    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20775493-c504-4316-9d6f-816952c83e9a-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.767571 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.767551    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflch\" (UniqueName: \"kubernetes.io/projected/20775493-c504-4316-9d6f-816952c83e9a-kube-api-access-rflch\") pod \"maas-default-gateway-openshift-default-694db55c87-hfxrj\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:24.881549 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:24.881514    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:25.309346 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:25.309266    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"]
Oct 30 21:52:25.348795 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:52:25.348767    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20775493_c504_4316_9d6f_816952c83e9a.slice/crio-b81d16b91da0c075c657a9873af7d26b6ceeb862183a17b22d5ffec00704f210 WatchSource:0}: Error finding container b81d16b91da0c075c657a9873af7d26b6ceeb862183a17b22d5ffec00704f210: Status 404 returned error can't find the container with id b81d16b91da0c075c657a9873af7d26b6ceeb862183a17b22d5ffec00704f210
Oct 30 21:52:25.353749 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:25.353629    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:52:25.353907 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:25.353838    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:52:25.354003 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:25.353923    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:52:25.990717 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:25.990679    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj" event={"ID":"20775493-c504-4316-9d6f-816952c83e9a","Type":"ContainerStarted","Data":"1ba57b2b607d4a4ec2a6c0a409d6c24898fcc8a932d16d197dd41e8999182789"}
Oct 30 21:52:25.991172 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:25.990747    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj" event={"ID":"20775493-c504-4316-9d6f-816952c83e9a","Type":"ContainerStarted","Data":"b81d16b91da0c075c657a9873af7d26b6ceeb862183a17b22d5ffec00704f210"}
Oct 30 21:52:26.229161 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:26.229053    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj" podStartSLOduration=2.229034064 podStartE2EDuration="2.229034064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:26.160922269 +0000 UTC m=+596.895285303" watchObservedRunningTime="2026-10-30 21:52:26.229034064 +0000 UTC m=+596.963397096"
Oct 30 21:52:26.882346 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:26.882288    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:26.884035 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:26.884006    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hfxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 21:52:26.884199 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:26.884076    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj" podUID="20775493-c504-4316-9d6f-816952c83e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 21:52:27.882713 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:27.882666    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hfxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 21:52:27.883482 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:27.882732    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj" podUID="20775493-c504-4316-9d6f-816952c83e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 21:52:28.201353 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:52:28.201267    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:52:28.882807 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:28.882774    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hfxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 21:52:28.883307 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:28.882831    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj" podUID="20775493-c504-4316-9d6f-816952c83e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 21:52:29.458777 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:29.458746    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 21:52:29.461411 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:29.461388    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 21:52:29.882615 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:29.882578    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hfxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 21:52:29.882794 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:29.882645    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj" podUID="20775493-c504-4316-9d6f-816952c83e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 21:52:30.881851 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:30.881816    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hfxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 21:52:30.882289 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:30.881880    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj" podUID="20775493-c504-4316-9d6f-816952c83e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 21:52:31.882348 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:31.882302    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hfxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 21:52:31.882348 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:31.882354    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj" podUID="20775493-c504-4316-9d6f-816952c83e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 21:52:32.882871 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:32.882824    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hfxrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 21:52:32.883450 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:32.882877    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj" podUID="20775493-c504-4316-9d6f-816952c83e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 21:52:33.886442 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:33.886411    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:33.886923 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:33.886717    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:33.887717 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:52:33.887692    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 21:52:58.199991 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:52:58.199952    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:53:23.657246 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.657211    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"]
Oct 30 21:53:23.660870 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.660853    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.741019 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.740976    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"]
Oct 30 21:53:23.766590 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.766559    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c6317794-gw-openshift-default-dockercfg-5hjdd\""
Oct 30 21:53:23.790846 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.790810    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-istio-envoy\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.791048 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.790891    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-istio-data\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.791048 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.790973    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-workload-certs\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.791201 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.791081    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-credential-socket\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.791201 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.791122    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be728331-4037-4264-8ed4-bddf8968a983-istio-podinfo\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.791298 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.791206    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-workload-socket\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.791298 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.791242    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54str\" (UniqueName: \"kubernetes.io/projected/be728331-4037-4264-8ed4-bddf8968a983-kube-api-access-54str\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.791384 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.791343    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be728331-4037-4264-8ed4-bddf8968a983-istio-token\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.791460 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.791445    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be728331-4037-4264-8ed4-bddf8968a983-istiod-ca-cert\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.892315 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.892277    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be728331-4037-4264-8ed4-bddf8968a983-istiod-ca-cert\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.892315 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.892317    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-istio-envoy\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.892541 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.892514    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-istio-data\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.892582 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.892568    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-workload-certs\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.892679 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.892657    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-credential-socket\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.892830 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.892692    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be728331-4037-4264-8ed4-bddf8968a983-istio-podinfo\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.892830 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.892722    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-workload-socket\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.892830 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.892752    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54str\" (UniqueName: \"kubernetes.io/projected/be728331-4037-4264-8ed4-bddf8968a983-kube-api-access-54str\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.892830 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.892779    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be728331-4037-4264-8ed4-bddf8968a983-istio-token\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.893228 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.892863    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-istio-data\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.893228 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.892919    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-workload-certs\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.893228 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.892960    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-credential-socket\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.893228 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.893186    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-workload-socket\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.893228 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.893216    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be728331-4037-4264-8ed4-bddf8968a983-istiod-ca-cert\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.894824 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.894802    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-istio-envoy\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.895250 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.895233    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be728331-4037-4264-8ed4-bddf8968a983-istio-podinfo\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.976042 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.975966    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be728331-4037-4264-8ed4-bddf8968a983-istio-token\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.976197 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.976150    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54str\" (UniqueName: \"kubernetes.io/projected/be728331-4037-4264-8ed4-bddf8968a983-kube-api-access-54str\") pod \"e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") " pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:23.982657 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:23.982633    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:24.433586 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:24.433559    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"]
Oct 30 21:53:24.435898 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:53:24.435870    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe728331_4037_4264_8ed4_bddf8968a983.slice/crio-075969fe2c9c80673566e49d72d29f223ee3f481555f61845e58919315cc8d3f WatchSource:0}: Error finding container 075969fe2c9c80673566e49d72d29f223ee3f481555f61845e58919315cc8d3f: Status 404 returned error can't find the container with id 075969fe2c9c80673566e49d72d29f223ee3f481555f61845e58919315cc8d3f
Oct 30 21:53:24.438053 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:24.438025    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:53:24.438120 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:24.438093    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:53:24.438195 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:24.438122    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:53:25.284216 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.284183    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr" event={"ID":"be728331-4037-4264-8ed4-bddf8968a983","Type":"ContainerStarted","Data":"1b03144bb745a654501af30a4afbffccb1b096913e4f1d61af346046e6dd1b64"}
Oct 30 21:53:25.284663 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.284237    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr" event={"ID":"be728331-4037-4264-8ed4-bddf8968a983","Type":"ContainerStarted","Data":"075969fe2c9c80673566e49d72d29f223ee3f481555f61845e58919315cc8d3f"}
Oct 30 21:53:25.288954 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.288926    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"]
Oct 30 21:53:25.292576 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.292555    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.374371 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.374342    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6996ffc6-openshift-default-dockercfg-mdrq6\""
Oct 30 21:53:25.408281 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.408252    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59361078-2381-4aef-9af4-46b6fdb7c391-istiod-ca-cert\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.408462 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.408292    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-workload-socket\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.408462 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.408377    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k78cp\" (UniqueName: \"kubernetes.io/projected/59361078-2381-4aef-9af4-46b6fdb7c391-kube-api-access-k78cp\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.408536 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.408484    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-credential-socket\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.408581 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.408550    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59361078-2381-4aef-9af4-46b6fdb7c391-istio-token\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.408615 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.408583    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-istio-envoy\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.408680 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.408660    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-istio-data\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.408766 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.408737    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59361078-2381-4aef-9af4-46b6fdb7c391-istio-podinfo\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.408821 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.408806    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-workload-certs\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.436639 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.436517    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr" podStartSLOduration=2.436494192 podStartE2EDuration="2.436494192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:25.434766188 +0000 UTC m=+656.169129213" watchObservedRunningTime="2026-10-30 21:53:25.436494192 +0000 UTC m=+656.170857225"
Oct 30 21:53:25.446009 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.445905    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"]
Oct 30 21:53:25.509612 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.509577    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-credential-socket\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.509810 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.509630    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59361078-2381-4aef-9af4-46b6fdb7c391-istio-token\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.509810 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.509662    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-istio-envoy\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.509998 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.509830    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-istio-data\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.509998 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.509863    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59361078-2381-4aef-9af4-46b6fdb7c391-istio-podinfo\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.509998 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.509892    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-workload-certs\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.509998 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.509986    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59361078-2381-4aef-9af4-46b6fdb7c391-istiod-ca-cert\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.510180 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.510024    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-credential-socket\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.510180 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.510030    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-workload-socket\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.510180 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.510107    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k78cp\" (UniqueName: \"kubernetes.io/projected/59361078-2381-4aef-9af4-46b6fdb7c391-kube-api-access-k78cp\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.510353 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.510333    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-workload-socket\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.511198 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.511165    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-workload-certs\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.511480 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.511459    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-istio-data\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.511725 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.511704    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59361078-2381-4aef-9af4-46b6fdb7c391-istiod-ca-cert\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.513316 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.513296    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-istio-envoy\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.513557 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.513541    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59361078-2381-4aef-9af4-46b6fdb7c391-istio-podinfo\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.576514 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.576438    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59361078-2381-4aef-9af4-46b6fdb7c391-istio-token\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.576514 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.576469    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k78cp\" (UniqueName: \"kubernetes.io/projected/59361078-2381-4aef-9af4-46b6fdb7c391-kube-api-access-k78cp\") pod \"e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") " pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.605909 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.605882    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:25.983448 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.983418    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:25.988751 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:25.988726    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:26.032369 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:26.032350    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"]
Oct 30 21:53:26.034318 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:53:26.034289    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59361078_2381_4aef_9af4_46b6fdb7c391.slice/crio-f4bcb55267b3a6939e914eda04d2a17f63609ea5039ccf9a6c13195a58c7fab6 WatchSource:0}: Error finding container f4bcb55267b3a6939e914eda04d2a17f63609ea5039ccf9a6c13195a58c7fab6: Status 404 returned error can't find the container with id f4bcb55267b3a6939e914eda04d2a17f63609ea5039ccf9a6c13195a58c7fab6
Oct 30 21:53:26.036676 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:26.036643    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:53:26.036785 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:26.036705    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:53:26.036785 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:26.036762    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:53:26.295119 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:26.295029    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm" event={"ID":"59361078-2381-4aef-9af4-46b6fdb7c391","Type":"ContainerStarted","Data":"ddfd092a6a3e845f7216f3d6aa7c42768ee2706cc02e0e9189ad2bc408170ea8"}
Oct 30 21:53:26.295119 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:26.295093    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm" event={"ID":"59361078-2381-4aef-9af4-46b6fdb7c391","Type":"ContainerStarted","Data":"f4bcb55267b3a6939e914eda04d2a17f63609ea5039ccf9a6c13195a58c7fab6"}
Oct 30 21:53:26.295593 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:26.295409    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:26.296508 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:26.296487    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:26.418873 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:26.418822    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm" podStartSLOduration=1.418808578 podStartE2EDuration="1.418808578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:26.416680763 +0000 UTC m=+657.151043795" watchObservedRunningTime="2026-10-30 21:53:26.418808578 +0000 UTC m=+657.153171609"
Oct 30 21:53:26.606126 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:26.606082    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:27.611999 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:27.611970    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:28.201303 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:53:28.201262    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:53:28.304809 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:28.304779    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:28.305648 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:28.305630    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:53:30.843296 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:30.843263    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"]
Oct 30 21:53:30.847231 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:30.847209    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"
Oct 30 21:53:30.893817 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:30.893784    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"]
Oct 30 21:53:30.942110 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:30.942064    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c6317794\""
Oct 30 21:53:30.942650 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:30.942630    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c6317794-dockercfg-fqtmx\""
Oct 30 21:53:30.967301 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:30.967266    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnqmk\" (UniqueName: \"kubernetes.io/projected/cc65e53d-cb8e-4149-9778-1955da5094d2-kube-api-access-vnqmk\") pod \"payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl\" (UID: \"cc65e53d-cb8e-4149-9778-1955da5094d2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"
Oct 30 21:53:30.967464 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:30.967427    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc65e53d-cb8e-4149-9778-1955da5094d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl\" (UID: \"cc65e53d-cb8e-4149-9778-1955da5094d2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"
Oct 30 21:53:30.994261 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:30.994218    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"]
Oct 30 21:53:30.997950 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:30.997928    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"
Oct 30 21:53:31.036796 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:31.036753    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"]
Oct 30 21:53:31.067967 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:31.067926    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc65e53d-cb8e-4149-9778-1955da5094d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl\" (UID: \"cc65e53d-cb8e-4149-9778-1955da5094d2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"
Oct 30 21:53:31.068201 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:31.067989    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnqmk\" (UniqueName: \"kubernetes.io/projected/cc65e53d-cb8e-4149-9778-1955da5094d2-kube-api-access-vnqmk\") pod \"payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl\" (UID: \"cc65e53d-cb8e-4149-9778-1955da5094d2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"
Oct 30 21:53:31.068687 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:31.068668    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc65e53d-cb8e-4149-9778-1955da5094d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl\" (UID: \"cc65e53d-cb8e-4149-9778-1955da5094d2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"
Oct 30 21:53:31.126733 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:31.126661    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnqmk\" (UniqueName: \"kubernetes.io/projected/cc65e53d-cb8e-4149-9778-1955da5094d2-kube-api-access-vnqmk\") pod \"payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl\" (UID: \"cc65e53d-cb8e-4149-9778-1955da5094d2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"
Oct 30 21:53:31.160550 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:31.160516    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"
Oct 30 21:53:31.168835 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:31.168796    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42l6l\" (UniqueName: \"kubernetes.io/projected/d6dc8f09-3dc8-4863-a741-4d61166dd247-kube-api-access-42l6l\") pod \"payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww\" (UID: \"d6dc8f09-3dc8-4863-a741-4d61166dd247\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"
Oct 30 21:53:31.168965 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:31.168940    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6dc8f09-3dc8-4863-a741-4d61166dd247-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww\" (UID: \"d6dc8f09-3dc8-4863-a741-4d61166dd247\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"
Oct 30 21:53:31.269828 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:31.269791    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42l6l\" (UniqueName: \"kubernetes.io/projected/d6dc8f09-3dc8-4863-a741-4d61166dd247-kube-api-access-42l6l\") pod \"payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww\" (UID: \"d6dc8f09-3dc8-4863-a741-4d61166dd247\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"
Oct 30 21:53:31.270003 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:31.269873    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6dc8f09-3dc8-4863-a741-4d61166dd247-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww\" (UID: \"d6dc8f09-3dc8-4863-a741-4d61166dd247\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"
Oct 30 21:53:31.270584 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:31.270565    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6dc8f09-3dc8-4863-a741-4d61166dd247-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww\" (UID: \"d6dc8f09-3dc8-4863-a741-4d61166dd247\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"
Oct 30 21:53:31.329118 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:31.329086    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42l6l\" (UniqueName: \"kubernetes.io/projected/d6dc8f09-3dc8-4863-a741-4d61166dd247-kube-api-access-42l6l\") pod \"payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww\" (UID: \"d6dc8f09-3dc8-4863-a741-4d61166dd247\") " pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"
Oct 30 21:53:31.541425 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:31.541400    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"]
Oct 30 21:53:31.543526 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:53:31.543498    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc65e53d_cb8e_4149_9778_1955da5094d2.slice/crio-74eba113d506eb3b3538d7ab973b7d4be6e62d985c39c9dda23eedaccd434180 WatchSource:0}: Error finding container 74eba113d506eb3b3538d7ab973b7d4be6e62d985c39c9dda23eedaccd434180: Status 404 returned error can't find the container with id 74eba113d506eb3b3538d7ab973b7d4be6e62d985c39c9dda23eedaccd434180
Oct 30 21:53:31.611424 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:31.611397    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"
Oct 30 21:53:31.985345 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:31.985321    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"]
Oct 30 21:53:31.986845 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:53:31.986816    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6dc8f09_3dc8_4863_a741_4d61166dd247.slice/crio-c3c6bcfa09ce06ede2be74b3fd5fa76d93933878c2c7e96d59345c6f1d89a3cf WatchSource:0}: Error finding container c3c6bcfa09ce06ede2be74b3fd5fa76d93933878c2c7e96d59345c6f1d89a3cf: Status 404 returned error can't find the container with id c3c6bcfa09ce06ede2be74b3fd5fa76d93933878c2c7e96d59345c6f1d89a3cf
Oct 30 21:53:32.324961 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:32.324858    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww" event={"ID":"d6dc8f09-3dc8-4863-a741-4d61166dd247","Type":"ContainerStarted","Data":"00dca65b8d78e1ee30c4145545ed34a87b3012bd5bb19cdaf52898fa6c9892e1"}
Oct 30 21:53:32.324961 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:32.324926    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww" event={"ID":"d6dc8f09-3dc8-4863-a741-4d61166dd247","Type":"ContainerStarted","Data":"c3c6bcfa09ce06ede2be74b3fd5fa76d93933878c2c7e96d59345c6f1d89a3cf"}
Oct 30 21:53:32.324961 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:32.324958    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"
Oct 30 21:53:32.326514 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:32.326493    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl" event={"ID":"cc65e53d-cb8e-4149-9778-1955da5094d2","Type":"ContainerStarted","Data":"d65d2e694f45658279fbec13e9656d6a60b21c2111d99e29456d88bf8ca378f0"}
Oct 30 21:53:32.326587 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:32.326533    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl" event={"ID":"cc65e53d-cb8e-4149-9778-1955da5094d2","Type":"ContainerStarted","Data":"74eba113d506eb3b3538d7ab973b7d4be6e62d985c39c9dda23eedaccd434180"}
Oct 30 21:53:32.326655 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:32.326614    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"
Oct 30 21:53:32.441154 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:32.441098    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww" podStartSLOduration=2.441083753 podStartE2EDuration="2.441083753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:32.439864847 +0000 UTC m=+663.174227879" watchObservedRunningTime="2026-10-30 21:53:32.441083753 +0000 UTC m=+663.175446783"
Oct 30 21:53:32.561181 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:32.561048    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl" podStartSLOduration=2.561029855 podStartE2EDuration="2.561029855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:32.559279591 +0000 UTC m=+663.293642645" watchObservedRunningTime="2026-10-30 21:53:32.561029855 +0000 UTC m=+663.295392888"
Oct 30 21:53:33.347640 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:33.347600    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"]
Oct 30 21:53:33.360907 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:33.360874    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:33.417035 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:33.417002    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"]
Oct 30 21:53:33.429166 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:33.429122    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c6317794-serving-cert\""
Oct 30 21:53:33.493899 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:33.493865    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-maas-api-tls\") pod \"maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:33.494379 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:33.494350    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/effe09b8-7438-4ca0-87b0-aaff35e43939-openshift-service-ca\") pod \"maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:33.494522 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:33.494458    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnsvp\" (UniqueName: \"kubernetes.io/projected/effe09b8-7438-4ca0-87b0-aaff35e43939-kube-api-access-dnsvp\") pod \"maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:33.494593 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:33.494556    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-metrics-certs\") pod \"maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:33.598779 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:33.598692    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-metrics-certs\") pod \"maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:33.599161 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:33.599120    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-maas-api-tls\") pod \"maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:33.599380 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:33.599363    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/effe09b8-7438-4ca0-87b0-aaff35e43939-openshift-service-ca\") pod \"maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:33.601083 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:33.601051    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnsvp\" (UniqueName: \"kubernetes.io/projected/effe09b8-7438-4ca0-87b0-aaff35e43939-kube-api-access-dnsvp\") pod \"maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:33.602195 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:53:33.602171    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-serving-cert: secret "maas-api-e2e-migrate-c6317794-serving-cert" not found
Oct 30 21:53:33.602323 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:53:33.602270    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-maas-api-tls podName:effe09b8-7438-4ca0-87b0-aaff35e43939 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:34.102245467 +0000 UTC m=+664.836608482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-maas-api-tls") pod "maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt" (UID: "effe09b8-7438-4ca0-87b0-aaff35e43939") : secret "maas-api-e2e-migrate-c6317794-serving-cert" not found
Oct 30 21:53:33.603503 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:33.602897    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/effe09b8-7438-4ca0-87b0-aaff35e43939-openshift-service-ca\") pod \"maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:33.605026 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:33.605000    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-metrics-certs\") pod \"maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:33.660496 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:33.660463    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnsvp\" (UniqueName: \"kubernetes.io/projected/effe09b8-7438-4ca0-87b0-aaff35e43939-kube-api-access-dnsvp\") pod \"maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:34.105952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:34.105924    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-maas-api-tls\") pod \"maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:34.108654 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:34.108622    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-maas-api-tls\") pod \"maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:34.275148 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:34.275092    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:34.650499 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:34.650471    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"]
Oct 30 21:53:34.652027 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:53:34.652002    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffe09b8_7438_4ca0_87b0_aaff35e43939.slice/crio-e3fcc3598e3db689c50efa570f250ef3b9e2e86f75fd31469e6b94c08e001868 WatchSource:0}: Error finding container e3fcc3598e3db689c50efa570f250ef3b9e2e86f75fd31469e6b94c08e001868: Status 404 returned error can't find the container with id e3fcc3598e3db689c50efa570f250ef3b9e2e86f75fd31469e6b94c08e001868
Oct 30 21:53:34.653520 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:34.653504    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:53:35.357340 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:35.357303    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt" event={"ID":"effe09b8-7438-4ca0-87b0-aaff35e43939","Type":"ContainerStarted","Data":"e3fcc3598e3db689c50efa570f250ef3b9e2e86f75fd31469e6b94c08e001868"}
Oct 30 21:53:35.785690 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:35.785655    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"]
Oct 30 21:53:35.789350 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:35.789335    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"
Oct 30 21:53:35.843682 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:35.843653    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f20e6e5e-dockercfg-7rcvv\""
Oct 30 21:53:35.849457 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:35.849420    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f20e6e5e\""
Oct 30 21:53:35.897531 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:35.897397    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"]
Oct 30 21:53:35.925888 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:35.925858    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gkv7\" (UniqueName: \"kubernetes.io/projected/7e29430b-0597-4d6a-9ecd-620ffc7a5904-kube-api-access-8gkv7\") pod \"payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs\" (UID: \"7e29430b-0597-4d6a-9ecd-620ffc7a5904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"
Oct 30 21:53:35.926077 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:35.925956    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e29430b-0597-4d6a-9ecd-620ffc7a5904-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs\" (UID: \"7e29430b-0597-4d6a-9ecd-620ffc7a5904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"
Oct 30 21:53:36.027380 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:36.027343    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gkv7\" (UniqueName: \"kubernetes.io/projected/7e29430b-0597-4d6a-9ecd-620ffc7a5904-kube-api-access-8gkv7\") pod \"payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs\" (UID: \"7e29430b-0597-4d6a-9ecd-620ffc7a5904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"
Oct 30 21:53:36.027531 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:36.027400    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e29430b-0597-4d6a-9ecd-620ffc7a5904-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs\" (UID: \"7e29430b-0597-4d6a-9ecd-620ffc7a5904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"
Oct 30 21:53:36.028085 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:36.028064    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e29430b-0597-4d6a-9ecd-620ffc7a5904-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs\" (UID: \"7e29430b-0597-4d6a-9ecd-620ffc7a5904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"
Oct 30 21:53:36.087007 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:36.086971    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gkv7\" (UniqueName: \"kubernetes.io/projected/7e29430b-0597-4d6a-9ecd-620ffc7a5904-kube-api-access-8gkv7\") pod \"payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs\" (UID: \"7e29430b-0597-4d6a-9ecd-620ffc7a5904\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"
Oct 30 21:53:36.102711 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:36.102684    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"
Oct 30 21:53:36.364276 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:36.364246    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt" event={"ID":"effe09b8-7438-4ca0-87b0-aaff35e43939","Type":"ContainerStarted","Data":"6cad21f92fbe43c0f129307013f6d153e7d4413f1fd42d6d74b5be41e60edae3"}
Oct 30 21:53:36.364428 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:36.364352    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:36.476159 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:36.476110    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"]
Oct 30 21:53:36.478277 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:53:36.478216    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e29430b_0597_4d6a_9ecd_620ffc7a5904.slice/crio-addf8c03e0ff38af3170f5133e5766dcf54820a17d2aa735909d6836874a2aee WatchSource:0}: Error finding container addf8c03e0ff38af3170f5133e5766dcf54820a17d2aa735909d6836874a2aee: Status 404 returned error can't find the container with id addf8c03e0ff38af3170f5133e5766dcf54820a17d2aa735909d6836874a2aee
Oct 30 21:53:36.488841 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:36.488799    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt" podStartSLOduration=2.546419921 podStartE2EDuration="3.488786126s" totalImagesPullingTime="942.366205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:36.487381619 +0000 UTC m=+667.221744652" watchObservedRunningTime="2026-10-30 21:53:36.488786126 +0000 UTC m=+667.223149157"
Oct 30 21:53:37.370589 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:37.370551    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs" event={"ID":"7e29430b-0597-4d6a-9ecd-620ffc7a5904","Type":"ContainerStarted","Data":"deb7350160b0456e3e764e088b4644cca312030dbc330cdc720d03302672382a"}
Oct 30 21:53:37.371058 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:37.370609    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs" event={"ID":"7e29430b-0597-4d6a-9ecd-620ffc7a5904","Type":"ContainerStarted","Data":"addf8c03e0ff38af3170f5133e5766dcf54820a17d2aa735909d6836874a2aee"}
Oct 30 21:53:37.371058 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:37.370731    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"
Oct 30 21:53:37.487661 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:37.487601    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs" podStartSLOduration=2.487585315 podStartE2EDuration="2.487585315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:37.486796225 +0000 UTC m=+668.221159255" watchObservedRunningTime="2026-10-30 21:53:37.487585315 +0000 UTC m=+668.221948345"
Oct 30 21:53:38.349603 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:38.349569    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"]
Oct 30 21:53:38.353256 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:38.353226    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:38.411482 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:38.411451    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f20e6e5e-serving-cert\""
Oct 30 21:53:38.453578 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:38.453539    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"]
Oct 30 21:53:38.455447 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:38.454813    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrmvz\" (UniqueName: \"kubernetes.io/projected/c120f97b-10f5-4df2-ad20-bd397b343ee4-kube-api-access-qrmvz\") pod \"maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:38.455447 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:38.454906    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-maas-api-tls\") pod \"maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:38.455447 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:38.454951    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-metrics-certs\") pod \"maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:38.455447 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:38.455127    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c120f97b-10f5-4df2-ad20-bd397b343ee4-openshift-service-ca\") pod \"maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:38.556178 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:38.556130    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrmvz\" (UniqueName: \"kubernetes.io/projected/c120f97b-10f5-4df2-ad20-bd397b343ee4-kube-api-access-qrmvz\") pod \"maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:38.556365 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:38.556236    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-maas-api-tls\") pod \"maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:38.556365 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:38.556260    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-metrics-certs\") pod \"maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:38.556365 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:38.556339    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c120f97b-10f5-4df2-ad20-bd397b343ee4-openshift-service-ca\") pod \"maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:38.556514 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:53:38.556377    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-serving-cert: secret "maas-api-e2e-ait-f20e6e5e-serving-cert" not found
Oct 30 21:53:38.556514 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:53:38.556444    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-maas-api-tls podName:c120f97b-10f5-4df2-ad20-bd397b343ee4 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:39.056420216 +0000 UTC m=+669.790783230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-maas-api-tls") pod "maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s" (UID: "c120f97b-10f5-4df2-ad20-bd397b343ee4") : secret "maas-api-e2e-ait-f20e6e5e-serving-cert" not found
Oct 30 21:53:38.557003 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:38.556983    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c120f97b-10f5-4df2-ad20-bd397b343ee4-openshift-service-ca\") pod \"maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:38.559450 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:38.559426    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-metrics-certs\") pod \"maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:38.617792 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:38.617719    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrmvz\" (UniqueName: \"kubernetes.io/projected/c120f97b-10f5-4df2-ad20-bd397b343ee4-kube-api-access-qrmvz\") pod \"maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:39.075258 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:39.075081    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-maas-api-tls\") pod \"maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:39.077850 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:39.077819    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-maas-api-tls\") pod \"maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:39.266073 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:39.266038    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:39.644629 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:39.644605    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"]
Oct 30 21:53:39.646389 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:53:39.646352    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc120f97b_10f5_4df2_ad20_bd397b343ee4.slice/crio-3cbc950e2d2579aad948f2cc56bde64e4fa1cccb2d6078f214c1197299745817 WatchSource:0}: Error finding container 3cbc950e2d2579aad948f2cc56bde64e4fa1cccb2d6078f214c1197299745817: Status 404 returned error can't find the container with id 3cbc950e2d2579aad948f2cc56bde64e4fa1cccb2d6078f214c1197299745817
Oct 30 21:53:40.385733 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:40.385691    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s" event={"ID":"c120f97b-10f5-4df2-ad20-bd397b343ee4","Type":"ContainerStarted","Data":"3cbc950e2d2579aad948f2cc56bde64e4fa1cccb2d6078f214c1197299745817"}
Oct 30 21:53:41.392208 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:41.392170    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s" event={"ID":"c120f97b-10f5-4df2-ad20-bd397b343ee4","Type":"ContainerStarted","Data":"9ab02ac91bc51b0f8a0701c81968578462153bea810d5fc3ea8d432668857fbb"}
Oct 30 21:53:41.392583 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:41.392227    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:41.529791 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:41.529724    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s" podStartSLOduration=2.646662009 podStartE2EDuration="3.529701141s" totalImagesPullingTime="883.039132ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:41.526586952 +0000 UTC m=+672.260949987" watchObservedRunningTime="2026-10-30 21:53:41.529701141 +0000 UTC m=+672.264064173"
Oct 30 21:53:42.376415 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:42.376384    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:53:43.343097 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:43.343062    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"
Oct 30 21:53:43.343517 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:43.343251    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"
Oct 30 21:53:47.402577 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:47.402551    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:53:48.379063 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:48.378991    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"
Oct 30 21:53:48.425606 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:48.425576    4553 generic.go:356] "Generic (PLEG): container finished" podID="be728331-4037-4264-8ed4-bddf8968a983" containerID="1b03144bb745a654501af30a4afbffccb1b096913e4f1d61af346046e6dd1b64" exitCode=0
Oct 30 21:53:48.426002 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:48.425648    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr" event={"ID":"be728331-4037-4264-8ed4-bddf8968a983","Type":"ContainerDied","Data":"1b03144bb745a654501af30a4afbffccb1b096913e4f1d61af346046e6dd1b64"}
Oct 30 21:53:48.426043 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:48.426012    4553 scope.go:122] "RemoveContainer" containerID="1b03144bb745a654501af30a4afbffccb1b096913e4f1d61af346046e6dd1b64"
Oct 30 21:53:48.427308 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:48.427281    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:53:48.427434 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:48.427361    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:53:48.427434 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:48.427407    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:53:48.817783 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:48.817698    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"]
Oct 30 21:53:48.821396 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:48.821373    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:48.875384 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:48.875345    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6996ffc6-serving-cert\""
Oct 30 21:53:48.881852 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:48.881816    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"]
Oct 30 21:53:48.903864 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:48.903821    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-metrics-certs\") pod \"maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:48.903864 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:48.903863    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69f95\" (UniqueName: \"kubernetes.io/projected/294adc13-f435-48c4-949d-778b66273652-kube-api-access-69f95\") pod \"maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:48.904128 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:48.904018    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-maas-api-tls\") pod \"maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:48.904128 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:48.904037    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/294adc13-f435-48c4-949d-778b66273652-openshift-service-ca\") pod \"maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:49.004632 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:49.004595    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-maas-api-tls\") pod \"maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:49.004632 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:49.004628    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/294adc13-f435-48c4-949d-778b66273652-openshift-service-ca\") pod \"maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:49.004893 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:49.004692    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-metrics-certs\") pod \"maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:49.004893 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:49.004713    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69f95\" (UniqueName: \"kubernetes.io/projected/294adc13-f435-48c4-949d-778b66273652-kube-api-access-69f95\") pod \"maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:49.004893 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:53:49.004750    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-serving-cert: secret "maas-api-e2e-ait-6996ffc6-serving-cert" not found
Oct 30 21:53:49.004893 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:53:49.004821    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-maas-api-tls podName:294adc13-f435-48c4-949d-778b66273652 nodeName:}" failed. No retries permitted until 2026-10-30 21:53:49.504799605 +0000 UTC m=+680.239162615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-maas-api-tls") pod "maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w" (UID: "294adc13-f435-48c4-949d-778b66273652") : secret "maas-api-e2e-ait-6996ffc6-serving-cert" not found
Oct 30 21:53:49.005385 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:49.005363    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/294adc13-f435-48c4-949d-778b66273652-openshift-service-ca\") pod \"maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:49.007446 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:49.007426    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-metrics-certs\") pod \"maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:49.067266 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:49.066669    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69f95\" (UniqueName: \"kubernetes.io/projected/294adc13-f435-48c4-949d-778b66273652-kube-api-access-69f95\") pod \"maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:49.438952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:49.438916    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr" event={"ID":"be728331-4037-4264-8ed4-bddf8968a983","Type":"ContainerStarted","Data":"b153506888d1fd8e47eea38cad1cfff0899fe5ccca151f01b83da9f6966cd13d"}
Oct 30 21:53:49.529158 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:49.527939    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-maas-api-tls\") pod \"maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:49.534649 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:49.534610    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-maas-api-tls\") pod \"maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:49.737283 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:49.737168    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:49.988713 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:49.988606    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:49.988896 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:49.988740    4553 patch_prober.go:29] interesting pod/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:53:49.988896 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:49.988795    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr" podUID="be728331-4037-4264-8ed4-bddf8968a983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 21:53:50.129160 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:50.128861    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"]
Oct 30 21:53:50.138274 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:53:50.138243    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294adc13_f435_48c4_949d_778b66273652.slice/crio-7827f652a31007a4954d1525e2f1df002e6aa24ea3ad61f5719410dea4716f8f WatchSource:0}: Error finding container 7827f652a31007a4954d1525e2f1df002e6aa24ea3ad61f5719410dea4716f8f: Status 404 returned error can't find the container with id 7827f652a31007a4954d1525e2f1df002e6aa24ea3ad61f5719410dea4716f8f
Oct 30 21:53:50.448733 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:50.448700    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w" event={"ID":"294adc13-f435-48c4-949d-778b66273652","Type":"ContainerStarted","Data":"7827f652a31007a4954d1525e2f1df002e6aa24ea3ad61f5719410dea4716f8f"}
Oct 30 21:53:50.983691 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:50.983665    4553 patch_prober.go:29] interesting pod/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:53:50.983804 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:50.983712    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr" podUID="be728331-4037-4264-8ed4-bddf8968a983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 21:53:51.455390 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:51.455356    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w" event={"ID":"294adc13-f435-48c4-949d-778b66273652","Type":"ContainerStarted","Data":"6252cee30802c68daadea51282588e307e6d08f7d0e3f96f76d11ea0d98fb750"}
Oct 30 21:53:51.455839 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:51.455472    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:51.588760 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:51.588709    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w" podStartSLOduration=2.751710418 podStartE2EDuration="3.588693354s" totalImagesPullingTime="836.982936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:51.587326243 +0000 UTC m=+682.321689275" watchObservedRunningTime="2026-10-30 21:53:51.588693354 +0000 UTC m=+682.323056414"
Oct 30 21:53:51.983801 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:51.983766    4553 patch_prober.go:29] interesting pod/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:53:51.983972 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:51.983827    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr" podUID="be728331-4037-4264-8ed4-bddf8968a983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 21:53:52.983774 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:52.983745    4553 patch_prober.go:29] interesting pod/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:53:52.984200 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:52.983794    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr" podUID="be728331-4037-4264-8ed4-bddf8968a983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 21:53:53.983430 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:53.983390    4553 patch_prober.go:29] interesting pod/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Oct 30 21:53:53.983737 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:53.983449    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr" podUID="be728331-4037-4264-8ed4-bddf8968a983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Oct 30 21:53:54.987476 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:54.987446    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:54.987958 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:54.987834    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:54.988571 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:54.988552    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:53:57.467546 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:57.467511    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:53:58.200926 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:53:58.200893    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:53:59.345018 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:59.344984    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"]
Oct 30 21:53:59.350281 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:59.350259    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:53:59.454206 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:59.454171    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-metrics-certs\") pod \"maas-api-e2e-ait-282df361-6594c5cc54-424hk\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:53:59.454387 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:59.454297    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3025e4a6-3016-41c0-9e08-a5c220cf3b54-openshift-service-ca\") pod \"maas-api-e2e-ait-282df361-6594c5cc54-424hk\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:53:59.454387 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:59.454342    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-maas-api-tls\") pod \"maas-api-e2e-ait-282df361-6594c5cc54-424hk\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:53:59.454581 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:59.454554    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-282df361-serving-cert\""
Oct 30 21:53:59.454641 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:59.454623    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg9mx\" (UniqueName: \"kubernetes.io/projected/3025e4a6-3016-41c0-9e08-a5c220cf3b54-kube-api-access-bg9mx\") pod \"maas-api-e2e-ait-282df361-6594c5cc54-424hk\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:53:59.456690 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:59.456669    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"]
Oct 30 21:53:59.556064 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:59.556022    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg9mx\" (UniqueName: \"kubernetes.io/projected/3025e4a6-3016-41c0-9e08-a5c220cf3b54-kube-api-access-bg9mx\") pod \"maas-api-e2e-ait-282df361-6594c5cc54-424hk\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:53:59.556277 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:59.556105    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-metrics-certs\") pod \"maas-api-e2e-ait-282df361-6594c5cc54-424hk\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:53:59.556277 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:59.556180    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3025e4a6-3016-41c0-9e08-a5c220cf3b54-openshift-service-ca\") pod \"maas-api-e2e-ait-282df361-6594c5cc54-424hk\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:53:59.556277 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:59.556225    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-maas-api-tls\") pod \"maas-api-e2e-ait-282df361-6594c5cc54-424hk\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:53:59.556455 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:53:59.556417    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-282df361-serving-cert: secret "maas-api-e2e-ait-282df361-serving-cert" not found
Oct 30 21:53:59.556510 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:53:59.556497    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-maas-api-tls podName:3025e4a6-3016-41c0-9e08-a5c220cf3b54 nodeName:}" failed. No retries permitted until 2026-10-30 21:54:00.05647487 +0000 UTC m=+690.790837903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-maas-api-tls") pod "maas-api-e2e-ait-282df361-6594c5cc54-424hk" (UID: "3025e4a6-3016-41c0-9e08-a5c220cf3b54") : secret "maas-api-e2e-ait-282df361-serving-cert" not found
Oct 30 21:53:59.557468 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:59.557437    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3025e4a6-3016-41c0-9e08-a5c220cf3b54-openshift-service-ca\") pod \"maas-api-e2e-ait-282df361-6594c5cc54-424hk\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:53:59.558831 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:59.558812    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-metrics-certs\") pod \"maas-api-e2e-ait-282df361-6594c5cc54-424hk\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:53:59.636991 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:53:59.636001    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg9mx\" (UniqueName: \"kubernetes.io/projected/3025e4a6-3016-41c0-9e08-a5c220cf3b54-kube-api-access-bg9mx\") pod \"maas-api-e2e-ait-282df361-6594c5cc54-424hk\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:54:00.063871 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:00.063241    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-maas-api-tls\") pod \"maas-api-e2e-ait-282df361-6594c5cc54-424hk\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:54:00.072429 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:00.072367    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-maas-api-tls\") pod \"maas-api-e2e-ait-282df361-6594c5cc54-424hk\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:54:00.265515 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:00.265475    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:54:00.669607 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:00.669584    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"]
Oct 30 21:54:00.671402 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:54:00.671376    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3025e4a6_3016_41c0_9e08_a5c220cf3b54.slice/crio-650eb3a702f9db287a00a30b78990dba63523ea75090e6c6949145a1605e96f7 WatchSource:0}: Error finding container 650eb3a702f9db287a00a30b78990dba63523ea75090e6c6949145a1605e96f7: Status 404 returned error can't find the container with id 650eb3a702f9db287a00a30b78990dba63523ea75090e6c6949145a1605e96f7
Oct 30 21:54:01.535161 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:01.535098    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk" event={"ID":"3025e4a6-3016-41c0-9e08-a5c220cf3b54","Type":"ContainerStarted","Data":"650eb3a702f9db287a00a30b78990dba63523ea75090e6c6949145a1605e96f7"}
Oct 30 21:54:02.542715 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:02.542679    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk" event={"ID":"3025e4a6-3016-41c0-9e08-a5c220cf3b54","Type":"ContainerStarted","Data":"d60afabfc88e7a408f6099d6b0490f7c4d7660bca9684d9c7eb1da21d2a3c488"}
Oct 30 21:54:02.543111 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:02.542821    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:54:02.660732 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:02.660679    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk" podStartSLOduration=2.737705781 podStartE2EDuration="3.660664017s" totalImagesPullingTime="922.958236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:02.659365484 +0000 UTC m=+693.393728516" watchObservedRunningTime="2026-10-30 21:54:02.660664017 +0000 UTC m=+693.395027091"
Oct 30 21:54:07.578354 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:07.578317    4553 generic.go:356] "Generic (PLEG): container finished" podID="59361078-2381-4aef-9af4-46b6fdb7c391" containerID="ddfd092a6a3e845f7216f3d6aa7c42768ee2706cc02e0e9189ad2bc408170ea8" exitCode=0
Oct 30 21:54:07.578820 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:07.578374    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm" event={"ID":"59361078-2381-4aef-9af4-46b6fdb7c391","Type":"ContainerDied","Data":"ddfd092a6a3e845f7216f3d6aa7c42768ee2706cc02e0e9189ad2bc408170ea8"}
Oct 30 21:54:07.578820 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:07.578751    4553 scope.go:122] "RemoveContainer" containerID="ddfd092a6a3e845f7216f3d6aa7c42768ee2706cc02e0e9189ad2bc408170ea8"
Oct 30 21:54:07.580192 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:07.580163    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:07.580259 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:07.580240    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:07.580294 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:07.580273    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:08.554249 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:08.554223    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:54:08.585824 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:08.585786    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm" event={"ID":"59361078-2381-4aef-9af4-46b6fdb7c391","Type":"ContainerStarted","Data":"71798f13cb744ec9d0e9462174df04643b4bb1ae77b2a280bbb93f50dc0b95bf"}
Oct 30 21:54:08.606292 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:08.606262    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:54:08.606575 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:08.606550    4553 patch_prober.go:29] interesting pod/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 21:54:08.606621 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:08.606602    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm" podUID="59361078-2381-4aef-9af4-46b6fdb7c391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 21:54:09.606566 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:09.606524    4553 patch_prober.go:29] interesting pod/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 21:54:09.606967 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:09.606592    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm" podUID="59361078-2381-4aef-9af4-46b6fdb7c391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 21:54:10.606896 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:10.606859    4553 patch_prober.go:29] interesting pod/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 21:54:10.607335 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:10.606926    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm" podUID="59361078-2381-4aef-9af4-46b6fdb7c391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 21:54:11.606915 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:11.606879    4553 patch_prober.go:29] interesting pod/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 21:54:11.607396 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:11.606933    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm" podUID="59361078-2381-4aef-9af4-46b6fdb7c391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 21:54:12.606552 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:12.606514    4553 patch_prober.go:29] interesting pod/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 21:54:12.606759 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:12.606566    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm" podUID="59361078-2381-4aef-9af4-46b6fdb7c391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 21:54:13.305231 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:13.305190    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:54:13.607085 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:13.607049    4553 patch_prober.go:29] interesting pod/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 21:54:13.607301 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:13.607112    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm" podUID="59361078-2381-4aef-9af4-46b6fdb7c391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 21:54:14.610608 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:14.610578    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:54:14.611675 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:14.611652    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:54:20.477732 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.477696    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"]
Oct 30 21:54:20.481614 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.481592    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:20.586300 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.586268    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c678f1ee-serving-cert\""
Oct 30 21:54:20.588342 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.588317    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"]
Oct 30 21:54:20.601277 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.601247    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncwnd\" (UniqueName: \"kubernetes.io/projected/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-kube-api-access-ncwnd\") pod \"maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:20.601402 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.601387    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-openshift-service-ca\") pod \"maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:20.601474 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.601459    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-maas-api-tls\") pod \"maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:20.601535 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.601521    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-metrics-certs\") pod \"maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:20.703098 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.703062    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-openshift-service-ca\") pod \"maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:20.703333 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.703201    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-maas-api-tls\") pod \"maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:20.703333 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.703236    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-metrics-certs\") pod \"maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:20.703333 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.703269    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncwnd\" (UniqueName: \"kubernetes.io/projected/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-kube-api-access-ncwnd\") pod \"maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:20.703894 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.703872    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-openshift-service-ca\") pod \"maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:20.705898 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.705874    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-maas-api-tls\") pod \"maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:20.705898 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.705885    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-metrics-certs\") pod \"maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:20.761530 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.761451    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncwnd\" (UniqueName: \"kubernetes.io/projected/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-kube-api-access-ncwnd\") pod \"maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:20.796407 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:20.796374    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:21.176776 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:21.176736    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"]
Oct 30 21:54:21.178746 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:54:21.178720    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04070c89_d0c2_4f45_8a25_e7a3944e3dfb.slice/crio-d4c7c48acfb7f2d4fa3eb11e216dd8b627534dafe7b57d1f9c41919fcac23794 WatchSource:0}: Error finding container d4c7c48acfb7f2d4fa3eb11e216dd8b627534dafe7b57d1f9c41919fcac23794: Status 404 returned error can't find the container with id d4c7c48acfb7f2d4fa3eb11e216dd8b627534dafe7b57d1f9c41919fcac23794
Oct 30 21:54:21.653061 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:21.653028    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b" event={"ID":"04070c89-d0c2-4f45-8a25-e7a3944e3dfb","Type":"ContainerStarted","Data":"d4c7c48acfb7f2d4fa3eb11e216dd8b627534dafe7b57d1f9c41919fcac23794"}
Oct 30 21:54:22.660126 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:22.660085    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b" event={"ID":"04070c89-d0c2-4f45-8a25-e7a3944e3dfb","Type":"ContainerStarted","Data":"7c296b6dae73027b9171c39697c2c461678b49a4abb2582c732cc6952acd872f"}
Oct 30 21:54:22.660589 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:22.660181    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:22.782197 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:22.782071    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b" podStartSLOduration=1.99044964 podStartE2EDuration="2.782055297s" totalImagesPullingTime="791.605657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:22.779094627 +0000 UTC m=+713.513457658" watchObservedRunningTime="2026-10-30 21:54:22.782055297 +0000 UTC m=+713.516418349"
Oct 30 21:54:25.406365 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.406333    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"]
Oct 30 21:54:25.406940 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.406524    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt" podUID="effe09b8-7438-4ca0-87b0-aaff35e43939" containerName="maas-api" containerID="cri-o://6cad21f92fbe43c0f129307013f6d153e7d4413f1fd42d6d74b5be41e60edae3" gracePeriod=30
Oct 30 21:54:25.676520 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.676494    4553 generic.go:356] "Generic (PLEG): container finished" podID="effe09b8-7438-4ca0-87b0-aaff35e43939" containerID="6cad21f92fbe43c0f129307013f6d153e7d4413f1fd42d6d74b5be41e60edae3" exitCode=0
Oct 30 21:54:25.676658 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.676580    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt" event={"ID":"effe09b8-7438-4ca0-87b0-aaff35e43939","Type":"ContainerDied","Data":"6cad21f92fbe43c0f129307013f6d153e7d4413f1fd42d6d74b5be41e60edae3"}
Oct 30 21:54:25.676658 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.676644    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt" event={"ID":"effe09b8-7438-4ca0-87b0-aaff35e43939","Type":"ContainerDied","Data":"e3fcc3598e3db689c50efa570f250ef3b9e2e86f75fd31469e6b94c08e001868"}
Oct 30 21:54:25.676729 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.676670    4553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3fcc3598e3db689c50efa570f250ef3b9e2e86f75fd31469e6b94c08e001868"
Oct 30 21:54:25.680966 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.680946    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:54:25.858973 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.858942    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-maas-api-tls\") pod \"effe09b8-7438-4ca0-87b0-aaff35e43939\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") "
Oct 30 21:54:25.859164 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.859060    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effe09b8-7438-4ca0-87b0-aaff35e43939-kube-api-access-dnsvp\" (UniqueName: \"kubernetes.io/projected/effe09b8-7438-4ca0-87b0-aaff35e43939-kube-api-access-dnsvp\") pod \"effe09b8-7438-4ca0-87b0-aaff35e43939\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") "
Oct 30 21:54:25.859164 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.859085    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-metrics-certs\" (UniqueName: \"kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-metrics-certs\") pod \"effe09b8-7438-4ca0-87b0-aaff35e43939\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") "
Oct 30 21:54:25.859164 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.859154    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/effe09b8-7438-4ca0-87b0-aaff35e43939-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/effe09b8-7438-4ca0-87b0-aaff35e43939-openshift-service-ca\") pod \"effe09b8-7438-4ca0-87b0-aaff35e43939\" (UID: \"effe09b8-7438-4ca0-87b0-aaff35e43939\") "
Oct 30 21:54:25.859615 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.859540    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effe09b8-7438-4ca0-87b0-aaff35e43939-openshift-service-ca" pod "effe09b8-7438-4ca0-87b0-aaff35e43939" (UID: "effe09b8-7438-4ca0-87b0-aaff35e43939"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:25.861064 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.861040    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-maas-api-tls" pod "effe09b8-7438-4ca0-87b0-aaff35e43939" (UID: "effe09b8-7438-4ca0-87b0-aaff35e43939"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:25.861303 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.861286    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effe09b8-7438-4ca0-87b0-aaff35e43939-kube-api-access-dnsvp" pod "effe09b8-7438-4ca0-87b0-aaff35e43939" (UID: "effe09b8-7438-4ca0-87b0-aaff35e43939"). InnerVolumeSpecName "kube-api-access-dnsvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:25.861369 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.861312    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-metrics-certs" pod "effe09b8-7438-4ca0-87b0-aaff35e43939" (UID: "effe09b8-7438-4ca0-87b0-aaff35e43939"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:25.907255 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.907219    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"]
Oct 30 21:54:25.960516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.960408    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:25.960516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.960455    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnsvp\" (UniqueName: \"kubernetes.io/projected/effe09b8-7438-4ca0-87b0-aaff35e43939-kube-api-access-dnsvp\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:25.960516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.960465    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/effe09b8-7438-4ca0-87b0-aaff35e43939-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:25.960516 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.960479    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/effe09b8-7438-4ca0-87b0-aaff35e43939-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:25.964284 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:25.964252    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"]
Oct 30 21:54:26.681738 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:26.681705    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"
Oct 30 21:54:26.852176 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:26.852120    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"]
Oct 30 21:54:26.856278 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:26.856249    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c6317794-7fcf6dd8c9-bg8zt"]
Oct 30 21:54:27.392470 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:27.392438    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="effe09b8-7438-4ca0-87b0-aaff35e43939" path="/var/lib/kubelet/pods/effe09b8-7438-4ca0-87b0-aaff35e43939/volumes"
Oct 30 21:54:28.201208 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:54:28.201174    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:54:28.670897 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:28.670865    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:54:30.908358 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:30.908298    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww" podUID="d6dc8f09-3dc8-4863-a741-4d61166dd247" containerName="payload-processing" containerID="cri-o://00dca65b8d78e1ee30c4145545ed34a87b3012bd5bb19cdaf52898fa6c9892e1" gracePeriod=25
Oct 30 21:54:30.965594 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:30.965549    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl" podUID="cc65e53d-cb8e-4149-9778-1955da5094d2" containerName="payload-pre-processing" containerID="cri-o://d65d2e694f45658279fbec13e9656d6a60b21c2111d99e29456d88bf8ca378f0" gracePeriod=25
Oct 30 21:54:31.214206 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.214181    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"
Oct 30 21:54:31.241053 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.241027    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"
Oct 30 21:54:31.310045 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.310013    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6dc8f09-3dc8-4863-a741-4d61166dd247-kube-api-access-42l6l\" (UniqueName: \"kubernetes.io/projected/d6dc8f09-3dc8-4863-a741-4d61166dd247-kube-api-access-42l6l\") pod \"d6dc8f09-3dc8-4863-a741-4d61166dd247\" (UID: \"d6dc8f09-3dc8-4863-a741-4d61166dd247\") "
Oct 30 21:54:31.310242 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.310073    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc65e53d-cb8e-4149-9778-1955da5094d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc65e53d-cb8e-4149-9778-1955da5094d2-plugins-config-volume\") pod \"cc65e53d-cb8e-4149-9778-1955da5094d2\" (UID: \"cc65e53d-cb8e-4149-9778-1955da5094d2\") "
Oct 30 21:54:31.310242 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.310100    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc65e53d-cb8e-4149-9778-1955da5094d2-kube-api-access-vnqmk\" (UniqueName: \"kubernetes.io/projected/cc65e53d-cb8e-4149-9778-1955da5094d2-kube-api-access-vnqmk\") pod \"cc65e53d-cb8e-4149-9778-1955da5094d2\" (UID: \"cc65e53d-cb8e-4149-9778-1955da5094d2\") "
Oct 30 21:54:31.310242 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.310169    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6dc8f09-3dc8-4863-a741-4d61166dd247-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6dc8f09-3dc8-4863-a741-4d61166dd247-plugins-config-volume\") pod \"d6dc8f09-3dc8-4863-a741-4d61166dd247\" (UID: \"d6dc8f09-3dc8-4863-a741-4d61166dd247\") "
Oct 30 21:54:31.310563 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.310528    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc65e53d-cb8e-4149-9778-1955da5094d2-plugins-config-volume" pod "cc65e53d-cb8e-4149-9778-1955da5094d2" (UID: "cc65e53d-cb8e-4149-9778-1955da5094d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:31.310701 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.310603    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6dc8f09-3dc8-4863-a741-4d61166dd247-plugins-config-volume" pod "d6dc8f09-3dc8-4863-a741-4d61166dd247" (UID: "d6dc8f09-3dc8-4863-a741-4d61166dd247"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:31.312373 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.312355    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6dc8f09-3dc8-4863-a741-4d61166dd247-kube-api-access-42l6l" pod "d6dc8f09-3dc8-4863-a741-4d61166dd247" (UID: "d6dc8f09-3dc8-4863-a741-4d61166dd247"). InnerVolumeSpecName "kube-api-access-42l6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:31.312373 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.312364    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc65e53d-cb8e-4149-9778-1955da5094d2-kube-api-access-vnqmk" pod "cc65e53d-cb8e-4149-9778-1955da5094d2" (UID: "cc65e53d-cb8e-4149-9778-1955da5094d2"). InnerVolumeSpecName "kube-api-access-vnqmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:31.411487 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.411457    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42l6l\" (UniqueName: \"kubernetes.io/projected/d6dc8f09-3dc8-4863-a741-4d61166dd247-kube-api-access-42l6l\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.411638 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.411490    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc65e53d-cb8e-4149-9778-1955da5094d2-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.411638 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.411506    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnqmk\" (UniqueName: \"kubernetes.io/projected/cc65e53d-cb8e-4149-9778-1955da5094d2-kube-api-access-vnqmk\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.411638 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.411520    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6dc8f09-3dc8-4863-a741-4d61166dd247-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.714512 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.714471    4553 generic.go:356] "Generic (PLEG): container finished" podID="d6dc8f09-3dc8-4863-a741-4d61166dd247" containerID="00dca65b8d78e1ee30c4145545ed34a87b3012bd5bb19cdaf52898fa6c9892e1" exitCode=0
Oct 30 21:54:31.714717 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.714544    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"
Oct 30 21:54:31.714717 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.714551    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww" event={"ID":"d6dc8f09-3dc8-4863-a741-4d61166dd247","Type":"ContainerDied","Data":"00dca65b8d78e1ee30c4145545ed34a87b3012bd5bb19cdaf52898fa6c9892e1"}
Oct 30 21:54:31.714717 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.714630    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww" event={"ID":"d6dc8f09-3dc8-4863-a741-4d61166dd247","Type":"ContainerDied","Data":"c3c6bcfa09ce06ede2be74b3fd5fa76d93933878c2c7e96d59345c6f1d89a3cf"}
Oct 30 21:54:31.714717 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.714658    4553 scope.go:122] "RemoveContainer" containerID="00dca65b8d78e1ee30c4145545ed34a87b3012bd5bb19cdaf52898fa6c9892e1"
Oct 30 21:54:31.716104 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.716071    4553 generic.go:356] "Generic (PLEG): container finished" podID="cc65e53d-cb8e-4149-9778-1955da5094d2" containerID="d65d2e694f45658279fbec13e9656d6a60b21c2111d99e29456d88bf8ca378f0" exitCode=0
Oct 30 21:54:31.716235 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.716150    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl" event={"ID":"cc65e53d-cb8e-4149-9778-1955da5094d2","Type":"ContainerDied","Data":"d65d2e694f45658279fbec13e9656d6a60b21c2111d99e29456d88bf8ca378f0"}
Oct 30 21:54:31.716235 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.716169    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"
Oct 30 21:54:31.716235 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.716200    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl" event={"ID":"cc65e53d-cb8e-4149-9778-1955da5094d2","Type":"ContainerDied","Data":"74eba113d506eb3b3538d7ab973b7d4be6e62d985c39c9dda23eedaccd434180"}
Oct 30 21:54:31.723944 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.723927    4553 scope.go:122] "RemoveContainer" containerID="00dca65b8d78e1ee30c4145545ed34a87b3012bd5bb19cdaf52898fa6c9892e1"
Oct 30 21:54:31.724263 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:54:31.724242    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00dca65b8d78e1ee30c4145545ed34a87b3012bd5bb19cdaf52898fa6c9892e1\": container with ID starting with 00dca65b8d78e1ee30c4145545ed34a87b3012bd5bb19cdaf52898fa6c9892e1 not found: ID does not exist" containerID="00dca65b8d78e1ee30c4145545ed34a87b3012bd5bb19cdaf52898fa6c9892e1"
Oct 30 21:54:31.724333 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.724272    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00dca65b8d78e1ee30c4145545ed34a87b3012bd5bb19cdaf52898fa6c9892e1"} err="failed to get container status \"00dca65b8d78e1ee30c4145545ed34a87b3012bd5bb19cdaf52898fa6c9892e1\": rpc error: code = NotFound desc = could not find container \"00dca65b8d78e1ee30c4145545ed34a87b3012bd5bb19cdaf52898fa6c9892e1\": container with ID starting with 00dca65b8d78e1ee30c4145545ed34a87b3012bd5bb19cdaf52898fa6c9892e1 not found: ID does not exist"
Oct 30 21:54:31.724333 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.724291    4553 scope.go:122] "RemoveContainer" containerID="d65d2e694f45658279fbec13e9656d6a60b21c2111d99e29456d88bf8ca378f0"
Oct 30 21:54:31.732822 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.732794    4553 scope.go:122] "RemoveContainer" containerID="d65d2e694f45658279fbec13e9656d6a60b21c2111d99e29456d88bf8ca378f0"
Oct 30 21:54:31.733157 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:54:31.733096    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d65d2e694f45658279fbec13e9656d6a60b21c2111d99e29456d88bf8ca378f0\": container with ID starting with d65d2e694f45658279fbec13e9656d6a60b21c2111d99e29456d88bf8ca378f0 not found: ID does not exist" containerID="d65d2e694f45658279fbec13e9656d6a60b21c2111d99e29456d88bf8ca378f0"
Oct 30 21:54:31.733260 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.733152    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d65d2e694f45658279fbec13e9656d6a60b21c2111d99e29456d88bf8ca378f0"} err="failed to get container status \"d65d2e694f45658279fbec13e9656d6a60b21c2111d99e29456d88bf8ca378f0\": rpc error: code = NotFound desc = could not find container \"d65d2e694f45658279fbec13e9656d6a60b21c2111d99e29456d88bf8ca378f0\": container with ID starting with d65d2e694f45658279fbec13e9656d6a60b21c2111d99e29456d88bf8ca378f0 not found: ID does not exist"
Oct 30 21:54:31.885075 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.885044    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"]
Oct 30 21:54:31.889410 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:31.889381    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c6317794-bf87cd95c-k76ww"]
Oct 30 21:54:32.000853 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:32.000764    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"]
Oct 30 21:54:32.006296 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:32.006268    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c6317794-964dbb7c9-s2mhl"]
Oct 30 21:54:33.392226 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:33.392198    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc65e53d-cb8e-4149-9778-1955da5094d2" path="/var/lib/kubelet/pods/cc65e53d-cb8e-4149-9778-1955da5094d2/volumes"
Oct 30 21:54:33.392606 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:33.392581    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6dc8f09-3dc8-4863-a741-4d61166dd247" path="/var/lib/kubelet/pods/d6dc8f09-3dc8-4863-a741-4d61166dd247/volumes"
Oct 30 21:54:34.774490 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:34.774458    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"]
Oct 30 21:54:34.774990 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:34.774681    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr" podUID="be728331-4037-4264-8ed4-bddf8968a983" containerName="istio-proxy" containerID="cri-o://b153506888d1fd8e47eea38cad1cfff0899fe5ccca151f01b83da9f6966cd13d" gracePeriod=30
Oct 30 21:54:36.138122 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.138062    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"]
Oct 30 21:54:36.138753 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.138679    4553 cpu_manager.go:414] "containerMap: removing container" podUID="cc65e53d-cb8e-4149-9778-1955da5094d2" containerName="payload-pre-processing"
Oct 30 21:54:36.138753 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.138697    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc65e53d-cb8e-4149-9778-1955da5094d2" containerName="payload-pre-processing"
Oct 30 21:54:36.138753 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.138727    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d6dc8f09-3dc8-4863-a741-4d61166dd247" containerName="payload-processing"
Oct 30 21:54:36.138753 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.138734    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6dc8f09-3dc8-4863-a741-4d61166dd247" containerName="payload-processing"
Oct 30 21:54:36.138753 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.138746    4553 cpu_manager.go:414] "containerMap: removing container" podUID="effe09b8-7438-4ca0-87b0-aaff35e43939" containerName="maas-api"
Oct 30 21:54:36.138753 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.138752    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="effe09b8-7438-4ca0-87b0-aaff35e43939" containerName="maas-api"
Oct 30 21:54:36.146484 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.139576    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc65e53d-cb8e-4149-9778-1955da5094d2" containerName="payload-pre-processing"
Oct 30 21:54:36.146484 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.139614    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6dc8f09-3dc8-4863-a741-4d61166dd247" containerName="payload-processing"
Oct 30 21:54:36.146484 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.139624    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="effe09b8-7438-4ca0-87b0-aaff35e43939" containerName="maas-api"
Oct 30 21:54:36.146484 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.145393    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.158968 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.158941    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.159093 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.158985    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.159093 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.159084    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.159215 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.159106    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.159215 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.159126    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.159215 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.159157    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.159376 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.159303    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.159376 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.159337    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwm8f\" (UniqueName: \"kubernetes.io/projected/f8f35f60-8e19-4353-85eb-73b8f7a36c11-kube-api-access-vwm8f\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.159471 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.159414    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.189995 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.189958    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"]
Oct 30 21:54:36.239094 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.239059    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"]
Oct 30 21:54:36.259975 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.259947    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.259975 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.259985    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.260274 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.260042    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.260274 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.260066    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwm8f\" (UniqueName: \"kubernetes.io/projected/f8f35f60-8e19-4353-85eb-73b8f7a36c11-kube-api-access-vwm8f\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.260274 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.260147    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.260274 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.260208    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.260274 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.260269    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.260522 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.260320    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.260522 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.260347    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.260522 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.260423    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.260522 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.260485    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.260791 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.260772    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.260869 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.260833    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.260869 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.260852    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.262796 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.262775    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.263110 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.263093    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.273795 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.273742    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-e90e35-openshift-default-dockercfg-4j6lz\""
Oct 30 21:54:36.319785 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.319759    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.320114 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.320078    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwm8f\" (UniqueName: \"kubernetes.io/projected/f8f35f60-8e19-4353-85eb-73b8f7a36c11-kube-api-access-vwm8f\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.459153 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.459039    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:36.935897 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.935866    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"]
Oct 30 21:54:36.938106 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:54:36.938078    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f35f60_8e19_4353_85eb_73b8f7a36c11.slice/crio-266fc4f1669c006635dc8c6716a52b206fec1e24f1f8bd7c71a5546d57dcfbf5 WatchSource:0}: Error finding container 266fc4f1669c006635dc8c6716a52b206fec1e24f1f8bd7c71a5546d57dcfbf5: Status 404 returned error can't find the container with id 266fc4f1669c006635dc8c6716a52b206fec1e24f1f8bd7c71a5546d57dcfbf5
Oct 30 21:54:36.940622 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.940592    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:36.940703 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.940660    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:36.940703 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:36.940688    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:37.763288 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:37.763243    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv" event={"ID":"f8f35f60-8e19-4353-85eb-73b8f7a36c11","Type":"ContainerStarted","Data":"5a94a89a74ee45e53e6272d40cfd852cd87b1b8ec1fd3adf065201de71690ac9"}
Oct 30 21:54:37.763841 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:37.763324    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv" event={"ID":"f8f35f60-8e19-4353-85eb-73b8f7a36c11","Type":"ContainerStarted","Data":"266fc4f1669c006635dc8c6716a52b206fec1e24f1f8bd7c71a5546d57dcfbf5"}
Oct 30 21:54:37.882937 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:37.882873    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv" podStartSLOduration=1.882853699 podStartE2EDuration="1.882853699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:37.8823754 +0000 UTC m=+728.616738447" watchObservedRunningTime="2026-10-30 21:54:37.882853699 +0000 UTC m=+728.617216733"
Oct 30 21:54:38.460250 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:38.460212    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:38.465694 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:38.465663    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:38.769220 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:38.769115    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:38.770089 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:38.770071    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:54:40.037479 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.037448    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:54:40.098258 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.098229    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be728331-4037-4264-8ed4-bddf8968a983-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be728331-4037-4264-8ed4-bddf8968a983-istiod-ca-cert\") pod \"be728331-4037-4264-8ed4-bddf8968a983\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") "
Oct 30 21:54:40.098461 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.098275    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-workload-socket\") pod \"be728331-4037-4264-8ed4-bddf8968a983\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") "
Oct 30 21:54:40.098461 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.098318    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-credential-socket\") pod \"be728331-4037-4264-8ed4-bddf8968a983\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") "
Oct 30 21:54:40.098461 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.098429    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be728331-4037-4264-8ed4-bddf8968a983-istio-token\" (UniqueName: \"kubernetes.io/projected/be728331-4037-4264-8ed4-bddf8968a983-istio-token\") pod \"be728331-4037-4264-8ed4-bddf8968a983\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") "
Oct 30 21:54:40.098642 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.098500    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be728331-4037-4264-8ed4-bddf8968a983-kube-api-access-54str\" (UniqueName: \"kubernetes.io/projected/be728331-4037-4264-8ed4-bddf8968a983-kube-api-access-54str\") pod \"be728331-4037-4264-8ed4-bddf8968a983\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") "
Oct 30 21:54:40.098642 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.098568    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-credential-socket" pod "be728331-4037-4264-8ed4-bddf8968a983" (UID: "be728331-4037-4264-8ed4-bddf8968a983"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:40.098642 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.098586    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-istio-envoy\") pod \"be728331-4037-4264-8ed4-bddf8968a983\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") "
Oct 30 21:54:40.098798 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.098634    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-workload-socket" pod "be728331-4037-4264-8ed4-bddf8968a983" (UID: "be728331-4037-4264-8ed4-bddf8968a983"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:40.098798 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.098641    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be728331-4037-4264-8ed4-bddf8968a983-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be728331-4037-4264-8ed4-bddf8968a983-istio-podinfo\") pod \"be728331-4037-4264-8ed4-bddf8968a983\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") "
Oct 30 21:54:40.098798 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.098750    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-workload-certs\") pod \"be728331-4037-4264-8ed4-bddf8968a983\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") "
Oct 30 21:54:40.098798 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.098766    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be728331-4037-4264-8ed4-bddf8968a983-istiod-ca-cert" pod "be728331-4037-4264-8ed4-bddf8968a983" (UID: "be728331-4037-4264-8ed4-bddf8968a983"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:40.099002 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.098805    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-istio-data\") pod \"be728331-4037-4264-8ed4-bddf8968a983\" (UID: \"be728331-4037-4264-8ed4-bddf8968a983\") "
Oct 30 21:54:40.099055 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.099038    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-workload-certs" pod "be728331-4037-4264-8ed4-bddf8968a983" (UID: "be728331-4037-4264-8ed4-bddf8968a983"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:40.099294 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.099239    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-istio-data" pod "be728331-4037-4264-8ed4-bddf8968a983" (UID: "be728331-4037-4264-8ed4-bddf8968a983"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:40.099294 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.099284    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be728331-4037-4264-8ed4-bddf8968a983-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.099466 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.099306    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.099466 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.099320    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.099466 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.099332    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.101096 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.101059    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be728331-4037-4264-8ed4-bddf8968a983-istio-token" pod "be728331-4037-4264-8ed4-bddf8968a983" (UID: "be728331-4037-4264-8ed4-bddf8968a983"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:40.101445 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.101418    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be728331-4037-4264-8ed4-bddf8968a983-istio-podinfo" pod "be728331-4037-4264-8ed4-bddf8968a983" (UID: "be728331-4037-4264-8ed4-bddf8968a983"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:40.101543 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.101511    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be728331-4037-4264-8ed4-bddf8968a983-kube-api-access-54str" pod "be728331-4037-4264-8ed4-bddf8968a983" (UID: "be728331-4037-4264-8ed4-bddf8968a983"). InnerVolumeSpecName "kube-api-access-54str". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:40.101619 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.101603    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-istio-envoy" pod "be728331-4037-4264-8ed4-bddf8968a983" (UID: "be728331-4037-4264-8ed4-bddf8968a983"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:40.200580 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.200547    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.200580 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.200572    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be728331-4037-4264-8ed4-bddf8968a983-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.200580 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.200582    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be728331-4037-4264-8ed4-bddf8968a983-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.200580 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.200591    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be728331-4037-4264-8ed4-bddf8968a983-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.200853 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.200601    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54str\" (UniqueName: \"kubernetes.io/projected/be728331-4037-4264-8ed4-bddf8968a983-kube-api-access-54str\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.782356 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.782326    4553 generic.go:356] "Generic (PLEG): container finished" podID="be728331-4037-4264-8ed4-bddf8968a983" containerID="b153506888d1fd8e47eea38cad1cfff0899fe5ccca151f01b83da9f6966cd13d" exitCode=0
Oct 30 21:54:40.782519 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.782389    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"
Oct 30 21:54:40.782519 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.782409    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr" event={"ID":"be728331-4037-4264-8ed4-bddf8968a983","Type":"ContainerDied","Data":"b153506888d1fd8e47eea38cad1cfff0899fe5ccca151f01b83da9f6966cd13d"}
Oct 30 21:54:40.782519 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.782502    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr" event={"ID":"be728331-4037-4264-8ed4-bddf8968a983","Type":"ContainerDied","Data":"075969fe2c9c80673566e49d72d29f223ee3f481555f61845e58919315cc8d3f"}
Oct 30 21:54:40.782666 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.782533    4553 scope.go:122] "RemoveContainer" containerID="b153506888d1fd8e47eea38cad1cfff0899fe5ccca151f01b83da9f6966cd13d"
Oct 30 21:54:40.792964 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.792944    4553 scope.go:122] "RemoveContainer" containerID="1b03144bb745a654501af30a4afbffccb1b096913e4f1d61af346046e6dd1b64"
Oct 30 21:54:40.801699 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.801678    4553 scope.go:122] "RemoveContainer" containerID="b153506888d1fd8e47eea38cad1cfff0899fe5ccca151f01b83da9f6966cd13d"
Oct 30 21:54:40.802002 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:54:40.801981    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b153506888d1fd8e47eea38cad1cfff0899fe5ccca151f01b83da9f6966cd13d\": container with ID starting with b153506888d1fd8e47eea38cad1cfff0899fe5ccca151f01b83da9f6966cd13d not found: ID does not exist" containerID="b153506888d1fd8e47eea38cad1cfff0899fe5ccca151f01b83da9f6966cd13d"
Oct 30 21:54:40.802051 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.802012    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b153506888d1fd8e47eea38cad1cfff0899fe5ccca151f01b83da9f6966cd13d"} err="failed to get container status \"b153506888d1fd8e47eea38cad1cfff0899fe5ccca151f01b83da9f6966cd13d\": rpc error: code = NotFound desc = could not find container \"b153506888d1fd8e47eea38cad1cfff0899fe5ccca151f01b83da9f6966cd13d\": container with ID starting with b153506888d1fd8e47eea38cad1cfff0899fe5ccca151f01b83da9f6966cd13d not found: ID does not exist"
Oct 30 21:54:40.802051 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.802031    4553 scope.go:122] "RemoveContainer" containerID="1b03144bb745a654501af30a4afbffccb1b096913e4f1d61af346046e6dd1b64"
Oct 30 21:54:40.802335 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:54:40.802317    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b03144bb745a654501af30a4afbffccb1b096913e4f1d61af346046e6dd1b64\": container with ID starting with 1b03144bb745a654501af30a4afbffccb1b096913e4f1d61af346046e6dd1b64 not found: ID does not exist" containerID="1b03144bb745a654501af30a4afbffccb1b096913e4f1d61af346046e6dd1b64"
Oct 30 21:54:40.802398 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.802341    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b03144bb745a654501af30a4afbffccb1b096913e4f1d61af346046e6dd1b64"} err="failed to get container status \"1b03144bb745a654501af30a4afbffccb1b096913e4f1d61af346046e6dd1b64\": rpc error: code = NotFound desc = could not find container \"1b03144bb745a654501af30a4afbffccb1b096913e4f1d61af346046e6dd1b64\": container with ID starting with 1b03144bb745a654501af30a4afbffccb1b096913e4f1d61af346046e6dd1b64 not found: ID does not exist"
Oct 30 21:54:40.989106 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.989070    4553 patch_prober.go:29] interesting pod/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:40.989335 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.989185    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr" podUID="be728331-4037-4264-8ed4-bddf8968a983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:40.997975 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:40.997949    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"]
Oct 30 21:54:41.048312 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.048237    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c6317794-gw-openshift-default-8585ff86fc-wtqrr"]
Oct 30 21:54:41.191354 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.191254    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs" podUID="7e29430b-0597-4d6a-9ecd-620ffc7a5904" containerName="payload-pre-processing" containerID="cri-o://deb7350160b0456e3e764e088b4644cca312030dbc330cdc720d03302672382a" gracePeriod=25
Oct 30 21:54:41.393539 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.393506    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be728331-4037-4264-8ed4-bddf8968a983" path="/var/lib/kubelet/pods/be728331-4037-4264-8ed4-bddf8968a983/volumes"
Oct 30 21:54:41.457712 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.457680    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"
Oct 30 21:54:41.514961 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.514929    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e29430b-0597-4d6a-9ecd-620ffc7a5904-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e29430b-0597-4d6a-9ecd-620ffc7a5904-plugins-config-volume\") pod \"7e29430b-0597-4d6a-9ecd-620ffc7a5904\" (UID: \"7e29430b-0597-4d6a-9ecd-620ffc7a5904\") "
Oct 30 21:54:41.515108 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.515040    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e29430b-0597-4d6a-9ecd-620ffc7a5904-kube-api-access-8gkv7\" (UniqueName: \"kubernetes.io/projected/7e29430b-0597-4d6a-9ecd-620ffc7a5904-kube-api-access-8gkv7\") pod \"7e29430b-0597-4d6a-9ecd-620ffc7a5904\" (UID: \"7e29430b-0597-4d6a-9ecd-620ffc7a5904\") "
Oct 30 21:54:41.515450 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.515421    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e29430b-0597-4d6a-9ecd-620ffc7a5904-plugins-config-volume" pod "7e29430b-0597-4d6a-9ecd-620ffc7a5904" (UID: "7e29430b-0597-4d6a-9ecd-620ffc7a5904"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:41.517329 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.517310    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e29430b-0597-4d6a-9ecd-620ffc7a5904-kube-api-access-8gkv7" pod "7e29430b-0597-4d6a-9ecd-620ffc7a5904" (UID: "7e29430b-0597-4d6a-9ecd-620ffc7a5904"). InnerVolumeSpecName "kube-api-access-8gkv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:41.616158 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.616098    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e29430b-0597-4d6a-9ecd-620ffc7a5904-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:41.616158 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.616157    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gkv7\" (UniqueName: \"kubernetes.io/projected/7e29430b-0597-4d6a-9ecd-620ffc7a5904-kube-api-access-8gkv7\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:41.789411 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.789379    4553 generic.go:356] "Generic (PLEG): container finished" podID="7e29430b-0597-4d6a-9ecd-620ffc7a5904" containerID="deb7350160b0456e3e764e088b4644cca312030dbc330cdc720d03302672382a" exitCode=0
Oct 30 21:54:41.789587 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.789427    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"
Oct 30 21:54:41.789587 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.789461    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs" event={"ID":"7e29430b-0597-4d6a-9ecd-620ffc7a5904","Type":"ContainerDied","Data":"deb7350160b0456e3e764e088b4644cca312030dbc330cdc720d03302672382a"}
Oct 30 21:54:41.789587 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.789513    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs" event={"ID":"7e29430b-0597-4d6a-9ecd-620ffc7a5904","Type":"ContainerDied","Data":"addf8c03e0ff38af3170f5133e5766dcf54820a17d2aa735909d6836874a2aee"}
Oct 30 21:54:41.789587 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.789530    4553 scope.go:122] "RemoveContainer" containerID="deb7350160b0456e3e764e088b4644cca312030dbc330cdc720d03302672382a"
Oct 30 21:54:41.798633 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.798606    4553 scope.go:122] "RemoveContainer" containerID="deb7350160b0456e3e764e088b4644cca312030dbc330cdc720d03302672382a"
Oct 30 21:54:41.798909 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:54:41.798889    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deb7350160b0456e3e764e088b4644cca312030dbc330cdc720d03302672382a\": container with ID starting with deb7350160b0456e3e764e088b4644cca312030dbc330cdc720d03302672382a not found: ID does not exist" containerID="deb7350160b0456e3e764e088b4644cca312030dbc330cdc720d03302672382a"
Oct 30 21:54:41.798969 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.798918    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb7350160b0456e3e764e088b4644cca312030dbc330cdc720d03302672382a"} err="failed to get container status \"deb7350160b0456e3e764e088b4644cca312030dbc330cdc720d03302672382a\": rpc error: code = NotFound desc = could not find container \"deb7350160b0456e3e764e088b4644cca312030dbc330cdc720d03302672382a\": container with ID starting with deb7350160b0456e3e764e088b4644cca312030dbc330cdc720d03302672382a not found: ID does not exist"
Oct 30 21:54:41.957498 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:41.957407    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"]
Oct 30 21:54:42.007786 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:42.007755    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f20e6e5e-586f7fd7f5-4hmrs"]
Oct 30 21:54:43.027887 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.027851    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"]
Oct 30 21:54:43.028323 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.028075    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w" podUID="294adc13-f435-48c4-949d-778b66273652" containerName="maas-api" containerID="cri-o://6252cee30802c68daadea51282588e307e6d08f7d0e3f96f76d11ea0d98fb750" gracePeriod=30
Oct 30 21:54:43.289780 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.289722    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:54:43.338891 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.338860    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294adc13-f435-48c4-949d-778b66273652-kube-api-access-69f95\" (UniqueName: \"kubernetes.io/projected/294adc13-f435-48c4-949d-778b66273652-kube-api-access-69f95\") pod \"294adc13-f435-48c4-949d-778b66273652\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") "
Oct 30 21:54:43.339037 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.338931    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-maas-api-tls\") pod \"294adc13-f435-48c4-949d-778b66273652\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") "
Oct 30 21:54:43.339037 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.338994    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/294adc13-f435-48c4-949d-778b66273652-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/294adc13-f435-48c4-949d-778b66273652-openshift-service-ca\") pod \"294adc13-f435-48c4-949d-778b66273652\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") "
Oct 30 21:54:43.339193 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.339077    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-metrics-certs\" (UniqueName: \"kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-metrics-certs\") pod \"294adc13-f435-48c4-949d-778b66273652\" (UID: \"294adc13-f435-48c4-949d-778b66273652\") "
Oct 30 21:54:43.339511 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.339482    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294adc13-f435-48c4-949d-778b66273652-openshift-service-ca" pod "294adc13-f435-48c4-949d-778b66273652" (UID: "294adc13-f435-48c4-949d-778b66273652"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:43.341371 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.341348    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294adc13-f435-48c4-949d-778b66273652-kube-api-access-69f95" pod "294adc13-f435-48c4-949d-778b66273652" (UID: "294adc13-f435-48c4-949d-778b66273652"). InnerVolumeSpecName "kube-api-access-69f95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:43.341465 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.341411    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-metrics-certs" pod "294adc13-f435-48c4-949d-778b66273652" (UID: "294adc13-f435-48c4-949d-778b66273652"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:43.341683 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.341661    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-maas-api-tls" pod "294adc13-f435-48c4-949d-778b66273652" (UID: "294adc13-f435-48c4-949d-778b66273652"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:43.397709 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.397676    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e29430b-0597-4d6a-9ecd-620ffc7a5904" path="/var/lib/kubelet/pods/7e29430b-0597-4d6a-9ecd-620ffc7a5904/volumes"
Oct 30 21:54:43.440734 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.440689    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:43.440734 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.440728    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/294adc13-f435-48c4-949d-778b66273652-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:43.440734 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.440742    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/294adc13-f435-48c4-949d-778b66273652-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:43.440996 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.440754    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69f95\" (UniqueName: \"kubernetes.io/projected/294adc13-f435-48c4-949d-778b66273652-kube-api-access-69f95\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:43.802894 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.802858    4553 generic.go:356] "Generic (PLEG): container finished" podID="294adc13-f435-48c4-949d-778b66273652" containerID="6252cee30802c68daadea51282588e307e6d08f7d0e3f96f76d11ea0d98fb750" exitCode=0
Oct 30 21:54:43.803080 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.802925    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"
Oct 30 21:54:43.803080 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.802948    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w" event={"ID":"294adc13-f435-48c4-949d-778b66273652","Type":"ContainerDied","Data":"6252cee30802c68daadea51282588e307e6d08f7d0e3f96f76d11ea0d98fb750"}
Oct 30 21:54:43.803080 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.803008    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w" event={"ID":"294adc13-f435-48c4-949d-778b66273652","Type":"ContainerDied","Data":"7827f652a31007a4954d1525e2f1df002e6aa24ea3ad61f5719410dea4716f8f"}
Oct 30 21:54:43.803080 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.803031    4553 scope.go:122] "RemoveContainer" containerID="6252cee30802c68daadea51282588e307e6d08f7d0e3f96f76d11ea0d98fb750"
Oct 30 21:54:43.812386 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.812366    4553 scope.go:122] "RemoveContainer" containerID="6252cee30802c68daadea51282588e307e6d08f7d0e3f96f76d11ea0d98fb750"
Oct 30 21:54:43.812713 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:54:43.812692    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6252cee30802c68daadea51282588e307e6d08f7d0e3f96f76d11ea0d98fb750\": container with ID starting with 6252cee30802c68daadea51282588e307e6d08f7d0e3f96f76d11ea0d98fb750 not found: ID does not exist" containerID="6252cee30802c68daadea51282588e307e6d08f7d0e3f96f76d11ea0d98fb750"
Oct 30 21:54:43.812789 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.812726    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6252cee30802c68daadea51282588e307e6d08f7d0e3f96f76d11ea0d98fb750"} err="failed to get container status \"6252cee30802c68daadea51282588e307e6d08f7d0e3f96f76d11ea0d98fb750\": rpc error: code = NotFound desc = could not find container \"6252cee30802c68daadea51282588e307e6d08f7d0e3f96f76d11ea0d98fb750\": container with ID starting with 6252cee30802c68daadea51282588e307e6d08f7d0e3f96f76d11ea0d98fb750 not found: ID does not exist"
Oct 30 21:54:43.968272 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.968236    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"]
Oct 30 21:54:43.972174 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:43.972148    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6996ffc6-69d46b7dc-fjz5w"]
Oct 30 21:54:44.717410 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.717377    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"]
Oct 30 21:54:44.717833 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.717820    4553 cpu_manager.go:414] "containerMap: removing container" podUID="be728331-4037-4264-8ed4-bddf8968a983" containerName="istio-proxy"
Oct 30 21:54:44.717880 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.717835    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be728331-4037-4264-8ed4-bddf8968a983" containerName="istio-proxy"
Oct 30 21:54:44.717880 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.717869    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7e29430b-0597-4d6a-9ecd-620ffc7a5904" containerName="payload-pre-processing"
Oct 30 21:54:44.717880 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.717875    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e29430b-0597-4d6a-9ecd-620ffc7a5904" containerName="payload-pre-processing"
Oct 30 21:54:44.717975 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.717882    4553 cpu_manager.go:414] "containerMap: removing container" podUID="294adc13-f435-48c4-949d-778b66273652" containerName="maas-api"
Oct 30 21:54:44.717975 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.717887    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294adc13-f435-48c4-949d-778b66273652" containerName="maas-api"
Oct 30 21:54:44.717975 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.717945    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="be728331-4037-4264-8ed4-bddf8968a983" containerName="istio-proxy"
Oct 30 21:54:44.717975 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.717955    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="be728331-4037-4264-8ed4-bddf8968a983" containerName="istio-proxy"
Oct 30 21:54:44.717975 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.717966    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e29430b-0597-4d6a-9ecd-620ffc7a5904" containerName="payload-pre-processing"
Oct 30 21:54:44.717975 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.717972    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="294adc13-f435-48c4-949d-778b66273652" containerName="maas-api"
Oct 30 21:54:44.722178 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.722162    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"
Oct 30 21:54:44.753907 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.753877    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f11f733e-3c5d-4fb2-b3fa-bbc206983605-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5\" (UID: \"f11f733e-3c5d-4fb2-b3fa-bbc206983605\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"
Oct 30 21:54:44.754070 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.753938    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btcfl\" (UniqueName: \"kubernetes.io/projected/f11f733e-3c5d-4fb2-b3fa-bbc206983605-kube-api-access-btcfl\") pod \"payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5\" (UID: \"f11f733e-3c5d-4fb2-b3fa-bbc206983605\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"
Oct 30 21:54:44.775970 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.775940    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-e90e35\""
Oct 30 21:54:44.779020 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.778993    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"]
Oct 30 21:54:44.839837 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.839794    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-e90e35-dockercfg-8kwsr\""
Oct 30 21:54:44.855966 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.855937    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btcfl\" (UniqueName: \"kubernetes.io/projected/f11f733e-3c5d-4fb2-b3fa-bbc206983605-kube-api-access-btcfl\") pod \"payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5\" (UID: \"f11f733e-3c5d-4fb2-b3fa-bbc206983605\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"
Oct 30 21:54:44.856340 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.856317    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f11f733e-3c5d-4fb2-b3fa-bbc206983605-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5\" (UID: \"f11f733e-3c5d-4fb2-b3fa-bbc206983605\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"
Oct 30 21:54:44.857054 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.857029    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f11f733e-3c5d-4fb2-b3fa-bbc206983605-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5\" (UID: \"f11f733e-3c5d-4fb2-b3fa-bbc206983605\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"
Oct 30 21:54:44.915814 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:44.915786    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btcfl\" (UniqueName: \"kubernetes.io/projected/f11f733e-3c5d-4fb2-b3fa-bbc206983605-kube-api-access-btcfl\") pod \"payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5\" (UID: \"f11f733e-3c5d-4fb2-b3fa-bbc206983605\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"
Oct 30 21:54:45.037995 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:45.037901    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"
Oct 30 21:54:45.392852 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:45.392813    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294adc13-f435-48c4-949d-778b66273652" path="/var/lib/kubelet/pods/294adc13-f435-48c4-949d-778b66273652/volumes"
Oct 30 21:54:45.411129 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:45.411107    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"]
Oct 30 21:54:45.412966 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:54:45.412939    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11f733e_3c5d_4fb2_b3fa_bbc206983605.slice/crio-65c0bc7078c0621a0d9e78551aa6816e9b58181d430b95f077d43293210b8819 WatchSource:0}: Error finding container 65c0bc7078c0621a0d9e78551aa6816e9b58181d430b95f077d43293210b8819: Status 404 returned error can't find the container with id 65c0bc7078c0621a0d9e78551aa6816e9b58181d430b95f077d43293210b8819
Oct 30 21:54:45.851941 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:45.851901    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5" event={"ID":"f11f733e-3c5d-4fb2-b3fa-bbc206983605","Type":"ContainerStarted","Data":"160a078ac404d75ab53630ef4d407e38bcfbcfdf98cc4a84d31e52fb22bf68a4"}
Oct 30 21:54:45.852369 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:45.851955    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5" event={"ID":"f11f733e-3c5d-4fb2-b3fa-bbc206983605","Type":"ContainerStarted","Data":"65c0bc7078c0621a0d9e78551aa6816e9b58181d430b95f077d43293210b8819"}
Oct 30 21:54:45.852369 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:45.852046    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"
Oct 30 21:54:45.974343 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:45.974291    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5" podStartSLOduration=1.974276272 podStartE2EDuration="1.974276272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:45.971476827 +0000 UTC m=+736.705839860" watchObservedRunningTime="2026-10-30 21:54:45.974276272 +0000 UTC m=+736.708639303"
Oct 30 21:54:46.024402 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:46.024370    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"]
Oct 30 21:54:46.024599 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:46.024579    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm" podUID="59361078-2381-4aef-9af4-46b6fdb7c391" containerName="istio-proxy" containerID="cri-o://71798f13cb744ec9d0e9462174df04643b4bb1ae77b2a280bbb93f50dc0b95bf" gracePeriod=30
Oct 30 21:54:48.093891 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.093853    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"]
Oct 30 21:54:48.094384 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.094368    4553 cpu_manager.go:414] "containerMap: removing container" podUID="be728331-4037-4264-8ed4-bddf8968a983" containerName="istio-proxy"
Oct 30 21:54:48.094449 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.094386    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be728331-4037-4264-8ed4-bddf8968a983" containerName="istio-proxy"
Oct 30 21:54:48.097682 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.097667    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.185695 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.185657    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b8717e-openshift-default-dockercfg-pqzrg\""
Oct 30 21:54:48.193146 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.193100    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.193333 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.193168    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.193333 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.193294    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzpw6\" (UniqueName: \"kubernetes.io/projected/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-kube-api-access-tzpw6\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.193442 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.193337    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.193442 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.193426    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.193535 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.193519    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.193589 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.193542    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.193638 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.193588    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.193686 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.193641    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.238512 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.238471    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"]
Oct 30 21:54:48.294980 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.294943    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzpw6\" (UniqueName: \"kubernetes.io/projected/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-kube-api-access-tzpw6\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.294980 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.294985    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.295276 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.295011    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.295276 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.295064    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.295276 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.295092    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.295462 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.295333    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.295462 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.295444    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.295567 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.295458    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.295567 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.295473    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.295567 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.295477    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.295567 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.295545    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.295753 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.295550    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.295753 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.295711    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.296117 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.296091    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.297942 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.297910    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.298170 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.298150    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.353712 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.353638    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.353848 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.353715    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzpw6\" (UniqueName: \"kubernetes.io/projected/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-kube-api-access-tzpw6\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.410187 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.410126    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:48.850421 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.850392    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"]
Oct 30 21:54:48.853098 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:54:48.853069    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2747809c_ab61_4e18_8cd3_e6bcfe0de5c3.slice/crio-7a7b3673de3c66d9b08d900dfedf0f9823a1ddf205fd6b5b20d0416e2519edfe WatchSource:0}: Error finding container 7a7b3673de3c66d9b08d900dfedf0f9823a1ddf205fd6b5b20d0416e2519edfe: Status 404 returned error can't find the container with id 7a7b3673de3c66d9b08d900dfedf0f9823a1ddf205fd6b5b20d0416e2519edfe
Oct 30 21:54:48.855743 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.855703    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:48.855848 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.855791    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:48.855848 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.855830    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:48.869608 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:48.869586    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b" event={"ID":"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3","Type":"ContainerStarted","Data":"7a7b3673de3c66d9b08d900dfedf0f9823a1ddf205fd6b5b20d0416e2519edfe"}
Oct 30 21:54:49.686080 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:49.686044    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"]
Oct 30 21:54:49.686540 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:49.686301    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk" podUID="3025e4a6-3016-41c0-9e08-a5c220cf3b54" containerName="maas-api" containerID="cri-o://d60afabfc88e7a408f6099d6b0490f7c4d7660bca9684d9c7eb1da21d2a3c488" gracePeriod=30
Oct 30 21:54:49.879625 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:49.879592    4553 generic.go:356] "Generic (PLEG): container finished" podID="3025e4a6-3016-41c0-9e08-a5c220cf3b54" containerID="d60afabfc88e7a408f6099d6b0490f7c4d7660bca9684d9c7eb1da21d2a3c488" exitCode=0
Oct 30 21:54:49.879770 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:49.879669    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk" event={"ID":"3025e4a6-3016-41c0-9e08-a5c220cf3b54","Type":"ContainerDied","Data":"d60afabfc88e7a408f6099d6b0490f7c4d7660bca9684d9c7eb1da21d2a3c488"}
Oct 30 21:54:49.881654 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:49.881626    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b" event={"ID":"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3","Type":"ContainerStarted","Data":"57b9223804e5c881c6c852272e96b288fd26968f286e3cf5e41f0687e6444dd7"}
Oct 30 21:54:49.962341 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:49.962310    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:54:50.002090 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.002039    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b" podStartSLOduration=2.002023552 podStartE2EDuration="2.002023552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:50.000452791 +0000 UTC m=+740.734815845" watchObservedRunningTime="2026-10-30 21:54:50.002023552 +0000 UTC m=+740.736386583"
Oct 30 21:54:50.030358 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.030327    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3025e4a6-3016-41c0-9e08-a5c220cf3b54-kube-api-access-bg9mx\" (UniqueName: \"kubernetes.io/projected/3025e4a6-3016-41c0-9e08-a5c220cf3b54-kube-api-access-bg9mx\") pod \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") "
Oct 30 21:54:50.030496 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.030387    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-maas-api-tls\") pod \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") "
Oct 30 21:54:50.030496 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.030416    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-metrics-certs\") pod \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") "
Oct 30 21:54:50.030496 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.030445    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3025e4a6-3016-41c0-9e08-a5c220cf3b54-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3025e4a6-3016-41c0-9e08-a5c220cf3b54-openshift-service-ca\") pod \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\" (UID: \"3025e4a6-3016-41c0-9e08-a5c220cf3b54\") "
Oct 30 21:54:50.030863 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.030840    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3025e4a6-3016-41c0-9e08-a5c220cf3b54-openshift-service-ca" pod "3025e4a6-3016-41c0-9e08-a5c220cf3b54" (UID: "3025e4a6-3016-41c0-9e08-a5c220cf3b54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:50.032747 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.032713    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-metrics-certs" pod "3025e4a6-3016-41c0-9e08-a5c220cf3b54" (UID: "3025e4a6-3016-41c0-9e08-a5c220cf3b54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:50.032747 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.032730    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-maas-api-tls" pod "3025e4a6-3016-41c0-9e08-a5c220cf3b54" (UID: "3025e4a6-3016-41c0-9e08-a5c220cf3b54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:50.032895 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.032820    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3025e4a6-3016-41c0-9e08-a5c220cf3b54-kube-api-access-bg9mx" pod "3025e4a6-3016-41c0-9e08-a5c220cf3b54" (UID: "3025e4a6-3016-41c0-9e08-a5c220cf3b54"). InnerVolumeSpecName "kube-api-access-bg9mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:50.132063 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.131957    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg9mx\" (UniqueName: \"kubernetes.io/projected/3025e4a6-3016-41c0-9e08-a5c220cf3b54-kube-api-access-bg9mx\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:50.132063 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.131995    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:50.132063 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.132010    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3025e4a6-3016-41c0-9e08-a5c220cf3b54-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:50.132063 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.132023    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3025e4a6-3016-41c0-9e08-a5c220cf3b54-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:50.410757 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.410721    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:50.415937 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.415900    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:50.887749 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.887722    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"
Oct 30 21:54:50.888242 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.887756    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk" event={"ID":"3025e4a6-3016-41c0-9e08-a5c220cf3b54","Type":"ContainerDied","Data":"650eb3a702f9db287a00a30b78990dba63523ea75090e6c6949145a1605e96f7"}
Oct 30 21:54:50.888242 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.887816    4553 scope.go:122] "RemoveContainer" containerID="d60afabfc88e7a408f6099d6b0490f7c4d7660bca9684d9c7eb1da21d2a3c488"
Oct 30 21:54:50.888368 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.888267    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:50.889428 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.889411    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:54:50.930057 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.930009    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"]
Oct 30 21:54:50.933547 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:50.930371    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s" podUID="c120f97b-10f5-4df2-ad20-bd397b343ee4" containerName="maas-api" containerID="cri-o://9ab02ac91bc51b0f8a0701c81968578462153bea810d5fc3ea8d432668857fbb" gracePeriod=30
Oct 30 21:54:51.235203 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.234320    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:54:51.244359 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.244328    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c120f97b-10f5-4df2-ad20-bd397b343ee4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c120f97b-10f5-4df2-ad20-bd397b343ee4-openshift-service-ca\") pod \"c120f97b-10f5-4df2-ad20-bd397b343ee4\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") "
Oct 30 21:54:51.244670 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.244648    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c120f97b-10f5-4df2-ad20-bd397b343ee4-openshift-service-ca" pod "c120f97b-10f5-4df2-ad20-bd397b343ee4" (UID: "c120f97b-10f5-4df2-ad20-bd397b343ee4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:51.244801 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.244787    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c120f97b-10f5-4df2-ad20-bd397b343ee4-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.297396 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.297373    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:54:51.311557 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.311530    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"]
Oct 30 21:54:51.315308 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.315283    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-282df361-6594c5cc54-424hk"]
Oct 30 21:54:51.354816 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.354775    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-istio-envoy\") pod \"59361078-2381-4aef-9af4-46b6fdb7c391\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") "
Oct 30 21:54:51.355020 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.354838    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-maas-api-tls\") pod \"c120f97b-10f5-4df2-ad20-bd397b343ee4\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") "
Oct 30 21:54:51.355020 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.354912    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59361078-2381-4aef-9af4-46b6fdb7c391-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59361078-2381-4aef-9af4-46b6fdb7c391-istio-podinfo\") pod \"59361078-2381-4aef-9af4-46b6fdb7c391\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") "
Oct 30 21:54:51.355020 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.354974    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-istio-data\") pod \"59361078-2381-4aef-9af4-46b6fdb7c391\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") "
Oct 30 21:54:51.355020 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.355006    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59361078-2381-4aef-9af4-46b6fdb7c391-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59361078-2381-4aef-9af4-46b6fdb7c391-istiod-ca-cert\") pod \"59361078-2381-4aef-9af4-46b6fdb7c391\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") "
Oct 30 21:54:51.355502 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.355043    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-workload-certs\") pod \"59361078-2381-4aef-9af4-46b6fdb7c391\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") "
Oct 30 21:54:51.355502 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.355100    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-credential-socket\") pod \"59361078-2381-4aef-9af4-46b6fdb7c391\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") "
Oct 30 21:54:51.355502 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.355223    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-workload-socket\") pod \"59361078-2381-4aef-9af4-46b6fdb7c391\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") "
Oct 30 21:54:51.355502 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.355284    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-metrics-certs\") pod \"c120f97b-10f5-4df2-ad20-bd397b343ee4\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") "
Oct 30 21:54:51.355502 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.355357    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59361078-2381-4aef-9af4-46b6fdb7c391-kube-api-access-k78cp\" (UniqueName: \"kubernetes.io/projected/59361078-2381-4aef-9af4-46b6fdb7c391-kube-api-access-k78cp\") pod \"59361078-2381-4aef-9af4-46b6fdb7c391\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") "
Oct 30 21:54:51.355502 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.355394    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59361078-2381-4aef-9af4-46b6fdb7c391-istio-token\" (UniqueName: \"kubernetes.io/projected/59361078-2381-4aef-9af4-46b6fdb7c391-istio-token\") pod \"59361078-2381-4aef-9af4-46b6fdb7c391\" (UID: \"59361078-2381-4aef-9af4-46b6fdb7c391\") "
Oct 30 21:54:51.355502 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.355450    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c120f97b-10f5-4df2-ad20-bd397b343ee4-kube-api-access-qrmvz\" (UniqueName: \"kubernetes.io/projected/c120f97b-10f5-4df2-ad20-bd397b343ee4-kube-api-access-qrmvz\") pod \"c120f97b-10f5-4df2-ad20-bd397b343ee4\" (UID: \"c120f97b-10f5-4df2-ad20-bd397b343ee4\") "
Oct 30 21:54:51.362260 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.359950    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-workload-certs" pod "59361078-2381-4aef-9af4-46b6fdb7c391" (UID: "59361078-2381-4aef-9af4-46b6fdb7c391"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:51.362260 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.359973    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-credential-socket" pod "59361078-2381-4aef-9af4-46b6fdb7c391" (UID: "59361078-2381-4aef-9af4-46b6fdb7c391"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:51.362260 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.360543    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c120f97b-10f5-4df2-ad20-bd397b343ee4-kube-api-access-qrmvz" pod "c120f97b-10f5-4df2-ad20-bd397b343ee4" (UID: "c120f97b-10f5-4df2-ad20-bd397b343ee4"). InnerVolumeSpecName "kube-api-access-qrmvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:51.362260 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.361607    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-workload-socket" pod "59361078-2381-4aef-9af4-46b6fdb7c391" (UID: "59361078-2381-4aef-9af4-46b6fdb7c391"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:51.365304 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.363530    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59361078-2381-4aef-9af4-46b6fdb7c391-istiod-ca-cert" pod "59361078-2381-4aef-9af4-46b6fdb7c391" (UID: "59361078-2381-4aef-9af4-46b6fdb7c391"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:51.365304 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.364478    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-istio-data" pod "59361078-2381-4aef-9af4-46b6fdb7c391" (UID: "59361078-2381-4aef-9af4-46b6fdb7c391"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:51.368220 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.368196    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-metrics-certs" pod "c120f97b-10f5-4df2-ad20-bd397b343ee4" (UID: "c120f97b-10f5-4df2-ad20-bd397b343ee4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:51.368704 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.368676    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59361078-2381-4aef-9af4-46b6fdb7c391-istio-podinfo" pod "59361078-2381-4aef-9af4-46b6fdb7c391" (UID: "59361078-2381-4aef-9af4-46b6fdb7c391"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:54:51.369008 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.368981    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59361078-2381-4aef-9af4-46b6fdb7c391-kube-api-access-k78cp" pod "59361078-2381-4aef-9af4-46b6fdb7c391" (UID: "59361078-2381-4aef-9af4-46b6fdb7c391"). InnerVolumeSpecName "kube-api-access-k78cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:51.370413 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.370386    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-istio-envoy" pod "59361078-2381-4aef-9af4-46b6fdb7c391" (UID: "59361078-2381-4aef-9af4-46b6fdb7c391"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:54:51.373514 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.373484    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59361078-2381-4aef-9af4-46b6fdb7c391-istio-token" pod "59361078-2381-4aef-9af4-46b6fdb7c391" (UID: "59361078-2381-4aef-9af4-46b6fdb7c391"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:51.374359 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.374176    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-maas-api-tls" pod "c120f97b-10f5-4df2-ad20-bd397b343ee4" (UID: "c120f97b-10f5-4df2-ad20-bd397b343ee4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:51.391854 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.391792    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3025e4a6-3016-41c0-9e08-a5c220cf3b54" path="/var/lib/kubelet/pods/3025e4a6-3016-41c0-9e08-a5c220cf3b54/volumes"
Oct 30 21:54:51.456880 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.456845    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59361078-2381-4aef-9af4-46b6fdb7c391-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.456880 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.456873    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.456880 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.456883    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59361078-2381-4aef-9af4-46b6fdb7c391-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.457101 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.456892    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.457101 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.456901    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.457101 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.456909    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.457101 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.456918    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.457101 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.456927    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k78cp\" (UniqueName: \"kubernetes.io/projected/59361078-2381-4aef-9af4-46b6fdb7c391-kube-api-access-k78cp\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.457101 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.456938    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59361078-2381-4aef-9af4-46b6fdb7c391-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.457101 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.456946    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrmvz\" (UniqueName: \"kubernetes.io/projected/c120f97b-10f5-4df2-ad20-bd397b343ee4-kube-api-access-qrmvz\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.457101 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.456957    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59361078-2381-4aef-9af4-46b6fdb7c391-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.457101 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.456966    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c120f97b-10f5-4df2-ad20-bd397b343ee4-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.894813 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.894772    4553 generic.go:356] "Generic (PLEG): container finished" podID="59361078-2381-4aef-9af4-46b6fdb7c391" containerID="71798f13cb744ec9d0e9462174df04643b4bb1ae77b2a280bbb93f50dc0b95bf" exitCode=0
Oct 30 21:54:51.895370 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.894854    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"
Oct 30 21:54:51.895370 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.894857    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm" event={"ID":"59361078-2381-4aef-9af4-46b6fdb7c391","Type":"ContainerDied","Data":"71798f13cb744ec9d0e9462174df04643b4bb1ae77b2a280bbb93f50dc0b95bf"}
Oct 30 21:54:51.895370 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.894997    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm" event={"ID":"59361078-2381-4aef-9af4-46b6fdb7c391","Type":"ContainerDied","Data":"f4bcb55267b3a6939e914eda04d2a17f63609ea5039ccf9a6c13195a58c7fab6"}
Oct 30 21:54:51.895370 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.895032    4553 scope.go:122] "RemoveContainer" containerID="71798f13cb744ec9d0e9462174df04643b4bb1ae77b2a280bbb93f50dc0b95bf"
Oct 30 21:54:51.896742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.896371    4553 generic.go:356] "Generic (PLEG): container finished" podID="c120f97b-10f5-4df2-ad20-bd397b343ee4" containerID="9ab02ac91bc51b0f8a0701c81968578462153bea810d5fc3ea8d432668857fbb" exitCode=0
Oct 30 21:54:51.896742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.896428    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"
Oct 30 21:54:51.896742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.896466    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s" event={"ID":"c120f97b-10f5-4df2-ad20-bd397b343ee4","Type":"ContainerDied","Data":"9ab02ac91bc51b0f8a0701c81968578462153bea810d5fc3ea8d432668857fbb"}
Oct 30 21:54:51.896742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.896510    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s" event={"ID":"c120f97b-10f5-4df2-ad20-bd397b343ee4","Type":"ContainerDied","Data":"3cbc950e2d2579aad948f2cc56bde64e4fa1cccb2d6078f214c1197299745817"}
Oct 30 21:54:51.905003 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.904983    4553 scope.go:122] "RemoveContainer" containerID="ddfd092a6a3e845f7216f3d6aa7c42768ee2706cc02e0e9189ad2bc408170ea8"
Oct 30 21:54:51.913435 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.913418    4553 scope.go:122] "RemoveContainer" containerID="71798f13cb744ec9d0e9462174df04643b4bb1ae77b2a280bbb93f50dc0b95bf"
Oct 30 21:54:51.913694 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:54:51.913674    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71798f13cb744ec9d0e9462174df04643b4bb1ae77b2a280bbb93f50dc0b95bf\": container with ID starting with 71798f13cb744ec9d0e9462174df04643b4bb1ae77b2a280bbb93f50dc0b95bf not found: ID does not exist" containerID="71798f13cb744ec9d0e9462174df04643b4bb1ae77b2a280bbb93f50dc0b95bf"
Oct 30 21:54:51.913784 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.913707    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71798f13cb744ec9d0e9462174df04643b4bb1ae77b2a280bbb93f50dc0b95bf"} err="failed to get container status \"71798f13cb744ec9d0e9462174df04643b4bb1ae77b2a280bbb93f50dc0b95bf\": rpc error: code = NotFound desc = could not find container \"71798f13cb744ec9d0e9462174df04643b4bb1ae77b2a280bbb93f50dc0b95bf\": container with ID starting with 71798f13cb744ec9d0e9462174df04643b4bb1ae77b2a280bbb93f50dc0b95bf not found: ID does not exist"
Oct 30 21:54:51.913784 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.913732    4553 scope.go:122] "RemoveContainer" containerID="ddfd092a6a3e845f7216f3d6aa7c42768ee2706cc02e0e9189ad2bc408170ea8"
Oct 30 21:54:51.913978 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:54:51.913962    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddfd092a6a3e845f7216f3d6aa7c42768ee2706cc02e0e9189ad2bc408170ea8\": container with ID starting with ddfd092a6a3e845f7216f3d6aa7c42768ee2706cc02e0e9189ad2bc408170ea8 not found: ID does not exist" containerID="ddfd092a6a3e845f7216f3d6aa7c42768ee2706cc02e0e9189ad2bc408170ea8"
Oct 30 21:54:51.914015 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.913984    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddfd092a6a3e845f7216f3d6aa7c42768ee2706cc02e0e9189ad2bc408170ea8"} err="failed to get container status \"ddfd092a6a3e845f7216f3d6aa7c42768ee2706cc02e0e9189ad2bc408170ea8\": rpc error: code = NotFound desc = could not find container \"ddfd092a6a3e845f7216f3d6aa7c42768ee2706cc02e0e9189ad2bc408170ea8\": container with ID starting with ddfd092a6a3e845f7216f3d6aa7c42768ee2706cc02e0e9189ad2bc408170ea8 not found: ID does not exist"
Oct 30 21:54:51.914015 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.914000    4553 scope.go:122] "RemoveContainer" containerID="9ab02ac91bc51b0f8a0701c81968578462153bea810d5fc3ea8d432668857fbb"
Oct 30 21:54:51.922053 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.922034    4553 scope.go:122] "RemoveContainer" containerID="9ab02ac91bc51b0f8a0701c81968578462153bea810d5fc3ea8d432668857fbb"
Oct 30 21:54:51.922332 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:54:51.922313    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab02ac91bc51b0f8a0701c81968578462153bea810d5fc3ea8d432668857fbb\": container with ID starting with 9ab02ac91bc51b0f8a0701c81968578462153bea810d5fc3ea8d432668857fbb not found: ID does not exist" containerID="9ab02ac91bc51b0f8a0701c81968578462153bea810d5fc3ea8d432668857fbb"
Oct 30 21:54:51.922376 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:51.922338    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab02ac91bc51b0f8a0701c81968578462153bea810d5fc3ea8d432668857fbb"} err="failed to get container status \"9ab02ac91bc51b0f8a0701c81968578462153bea810d5fc3ea8d432668857fbb\": rpc error: code = NotFound desc = could not find container \"9ab02ac91bc51b0f8a0701c81968578462153bea810d5fc3ea8d432668857fbb\": container with ID starting with 9ab02ac91bc51b0f8a0701c81968578462153bea810d5fc3ea8d432668857fbb not found: ID does not exist"
Oct 30 21:54:52.063173 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:52.063114    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"]
Oct 30 21:54:52.113335 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:52.113295    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6996ffc6-openshift-default-6f99ccc9c5-8hpvm"]
Oct 30 21:54:52.175432 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:52.175337    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"]
Oct 30 21:54:52.181364 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:52.181330    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f20e6e5e-54468fcfdd-9pg9s"]
Oct 30 21:54:53.391796 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:53.391753    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59361078-2381-4aef-9af4-46b6fdb7c391" path="/var/lib/kubelet/pods/59361078-2381-4aef-9af4-46b6fdb7c391/volumes"
Oct 30 21:54:53.392369 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:53.392350    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c120f97b-10f5-4df2-ad20-bd397b343ee4" path="/var/lib/kubelet/pods/c120f97b-10f5-4df2-ad20-bd397b343ee4/volumes"
Oct 30 21:54:56.860882 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:56.860854    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"
Oct 30 21:54:57.734362 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.734329    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"]
Oct 30 21:54:57.734844 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.734827    4553 cpu_manager.go:414] "containerMap: removing container" podUID="59361078-2381-4aef-9af4-46b6fdb7c391" containerName="istio-proxy"
Oct 30 21:54:57.734937 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.734848    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59361078-2381-4aef-9af4-46b6fdb7c391" containerName="istio-proxy"
Oct 30 21:54:57.734937 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.734871    4553 cpu_manager.go:414] "containerMap: removing container" podUID="3025e4a6-3016-41c0-9e08-a5c220cf3b54" containerName="maas-api"
Oct 30 21:54:57.734937 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.734878    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3025e4a6-3016-41c0-9e08-a5c220cf3b54" containerName="maas-api"
Oct 30 21:54:57.734937 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.734906    4553 cpu_manager.go:414] "containerMap: removing container" podUID="c120f97b-10f5-4df2-ad20-bd397b343ee4" containerName="maas-api"
Oct 30 21:54:57.734937 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.734914    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c120f97b-10f5-4df2-ad20-bd397b343ee4" containerName="maas-api"
Oct 30 21:54:57.735220 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.735027    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="59361078-2381-4aef-9af4-46b6fdb7c391" containerName="istio-proxy"
Oct 30 21:54:57.735220 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.735042    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="3025e4a6-3016-41c0-9e08-a5c220cf3b54" containerName="maas-api"
Oct 30 21:54:57.735220 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.735053    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="c120f97b-10f5-4df2-ad20-bd397b343ee4" containerName="maas-api"
Oct 30 21:54:57.740115 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.740094    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.795261 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.795234    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d2a3e0-openshift-default-dockercfg-t4vpk\""
Oct 30 21:54:57.798335 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.798306    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"]
Oct 30 21:54:57.838680 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.838645    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4zl\" (UniqueName: \"kubernetes.io/projected/5c189afe-afe8-4736-8e80-e321da9d7268-kube-api-access-mm4zl\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.838840 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.838686    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.838840 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.838754    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.838840 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.838776    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c189afe-afe8-4736-8e80-e321da9d7268-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.838840 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.838798    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.838840 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.838828    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c189afe-afe8-4736-8e80-e321da9d7268-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.839003 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.838881    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c189afe-afe8-4736-8e80-e321da9d7268-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.839003 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.838992    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.839070 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.839055    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.940570 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.940536    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.941044 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.940632    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4zl\" (UniqueName: \"kubernetes.io/projected/5c189afe-afe8-4736-8e80-e321da9d7268-kube-api-access-mm4zl\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.941044 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.940668    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.941044 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.940692    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.941044 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.940718    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c189afe-afe8-4736-8e80-e321da9d7268-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.941044 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.940813    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.941044 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.940883    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c189afe-afe8-4736-8e80-e321da9d7268-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.941044 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.940908    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c189afe-afe8-4736-8e80-e321da9d7268-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.941044 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.940954    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.941044 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.940991    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.941044 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.941025    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.941740 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.941069    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.941740 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.941437    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.941740 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.941650    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c189afe-afe8-4736-8e80-e321da9d7268-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.943328 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.943305    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:57.943328 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:57.943317    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c189afe-afe8-4736-8e80-e321da9d7268-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:58.000884 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.000797    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c189afe-afe8-4736-8e80-e321da9d7268-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:58.001450 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.001428    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4zl\" (UniqueName: \"kubernetes.io/projected/5c189afe-afe8-4736-8e80-e321da9d7268-kube-api-access-mm4zl\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:58.054350 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.054316    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:58.092573 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.092543    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"]
Oct 30 21:54:58.093048 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.093022    4553 cpu_manager.go:414] "containerMap: removing container" podUID="59361078-2381-4aef-9af4-46b6fdb7c391" containerName="istio-proxy"
Oct 30 21:54:58.093048 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.093039    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59361078-2381-4aef-9af4-46b6fdb7c391" containerName="istio-proxy"
Oct 30 21:54:58.093270 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.093183    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="59361078-2381-4aef-9af4-46b6fdb7c391" containerName="istio-proxy"
Oct 30 21:54:58.097624 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.097603    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:54:58.161914 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.161864    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"]
Oct 30 21:54:58.208985 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:54:58.208917    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:54:58.239381 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.239351    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-e90e35-serving-cert\""
Oct 30 21:54:58.244840 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.244818    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:54:58.244947 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.244887    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb47f460-f5f8-4f9f-908b-4c4532072b3f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:54:58.245050 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.245032    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k86k\" (UniqueName: \"kubernetes.io/projected/fb47f460-f5f8-4f9f-908b-4c4532072b3f-kube-api-access-2k86k\") pod \"maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:54:58.245192 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.245174    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-metrics-certs\") pod \"maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:54:58.346615 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.346583    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k86k\" (UniqueName: \"kubernetes.io/projected/fb47f460-f5f8-4f9f-908b-4c4532072b3f-kube-api-access-2k86k\") pod \"maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:54:58.346812 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.346665    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-metrics-certs\") pod \"maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:54:58.346812 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.346695    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:54:58.346938 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:54:58.346820    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-serving-cert: secret "maas-api-e2e-worker-w3-e90e35-serving-cert" not found
Oct 30 21:54:58.346938 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:54:58.346877    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-maas-api-tls podName:fb47f460-f5f8-4f9f-908b-4c4532072b3f nodeName:}" failed. No retries permitted until 2026-10-30 21:54:58.846857147 +0000 UTC m=+749.581220170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-maas-api-tls") pod "maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84" (UID: "fb47f460-f5f8-4f9f-908b-4c4532072b3f") : secret "maas-api-e2e-worker-w3-e90e35-serving-cert" not found
Oct 30 21:54:58.346938 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.346892    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb47f460-f5f8-4f9f-908b-4c4532072b3f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:54:58.347578 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.347554    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb47f460-f5f8-4f9f-908b-4c4532072b3f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:54:58.349793 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.349762    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-metrics-certs\") pod \"maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:54:58.404774 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.404747    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k86k\" (UniqueName: \"kubernetes.io/projected/fb47f460-f5f8-4f9f-908b-4c4532072b3f-kube-api-access-2k86k\") pod \"maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:54:58.486986 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.486962    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"]
Oct 30 21:54:58.489041 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:54:58.489016    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c189afe_afe8_4736_8e80_e321da9d7268.slice/crio-696b5c97f32851f6edff9beadc4b250f6ebdb59b22ca4f9efe693ad6023593e5 WatchSource:0}: Error finding container 696b5c97f32851f6edff9beadc4b250f6ebdb59b22ca4f9efe693ad6023593e5: Status 404 returned error can't find the container with id 696b5c97f32851f6edff9beadc4b250f6ebdb59b22ca4f9efe693ad6023593e5
Oct 30 21:54:58.491411 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.491380    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:58.491476 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.491442    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:58.491476 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.491472    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:58.862213 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.861796    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:54:58.885517 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.885447    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:54:58.939663 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.937194    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q" event={"ID":"5c189afe-afe8-4736-8e80-e321da9d7268","Type":"ContainerStarted","Data":"5db73e8a9d255a13bd7b722d14e7a376efd66a26fdf799304296c1b2f88040d0"}
Oct 30 21:54:58.939663 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:58.937302    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q" event={"ID":"5c189afe-afe8-4736-8e80-e321da9d7268","Type":"ContainerStarted","Data":"696b5c97f32851f6edff9beadc4b250f6ebdb59b22ca4f9efe693ad6023593e5"}
Oct 30 21:54:59.017374 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.017335    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:54:59.055283 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.055205    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:54:59.056826 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.056799    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:54:59.056957 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.056857    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q" podUID="5c189afe-afe8-4736-8e80-e321da9d7268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 21:54:59.065245 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.065200    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q" podStartSLOduration=2.065187588 podStartE2EDuration="2.065187588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:59.064687043 +0000 UTC m=+749.799050085" watchObservedRunningTime="2026-10-30 21:54:59.065187588 +0000 UTC m=+749.799550618"
Oct 30 21:54:59.137093 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.137021    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"]
Oct 30 21:54:59.141794 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.141774    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"
Oct 30 21:54:59.197924 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.197887    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"]
Oct 30 21:54:59.201780 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.201750    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b8717e\""
Oct 30 21:54:59.201964 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.201920    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b8717e-dockercfg-nklwb\""
Oct 30 21:54:59.265854 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.265811    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dtx6\" (UniqueName: \"kubernetes.io/projected/8f0e9ade-f4a6-4795-bda4-63df0eedf974-kube-api-access-4dtx6\") pod \"payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w\" (UID: \"8f0e9ade-f4a6-4795-bda4-63df0eedf974\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"
Oct 30 21:54:59.266050 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.265995    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0e9ade-f4a6-4795-bda4-63df0eedf974-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w\" (UID: \"8f0e9ade-f4a6-4795-bda4-63df0eedf974\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"
Oct 30 21:54:59.288572 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.288541    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"]
Oct 30 21:54:59.293467 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.293443    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"
Oct 30 21:54:59.349686 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.349649    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"]
Oct 30 21:54:59.367097 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.367047    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0e9ade-f4a6-4795-bda4-63df0eedf974-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w\" (UID: \"8f0e9ade-f4a6-4795-bda4-63df0eedf974\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"
Oct 30 21:54:59.367309 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.367098    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjn8\" (UniqueName: \"kubernetes.io/projected/d158a3d2-ee91-4ff9-8def-629768a484d2-kube-api-access-7kjn8\") pod \"payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72\" (UID: \"d158a3d2-ee91-4ff9-8def-629768a484d2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"
Oct 30 21:54:59.367309 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.367214    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dtx6\" (UniqueName: \"kubernetes.io/projected/8f0e9ade-f4a6-4795-bda4-63df0eedf974-kube-api-access-4dtx6\") pod \"payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w\" (UID: \"8f0e9ade-f4a6-4795-bda4-63df0eedf974\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"
Oct 30 21:54:59.367419 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.367367    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d158a3d2-ee91-4ff9-8def-629768a484d2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72\" (UID: \"d158a3d2-ee91-4ff9-8def-629768a484d2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"
Oct 30 21:54:59.367980 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.367959    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0e9ade-f4a6-4795-bda4-63df0eedf974-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w\" (UID: \"8f0e9ade-f4a6-4795-bda4-63df0eedf974\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"
Oct 30 21:54:59.427993 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.427920    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dtx6\" (UniqueName: \"kubernetes.io/projected/8f0e9ade-f4a6-4795-bda4-63df0eedf974-kube-api-access-4dtx6\") pod \"payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w\" (UID: \"8f0e9ade-f4a6-4795-bda4-63df0eedf974\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"
Oct 30 21:54:59.455523 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.455492    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"
Oct 30 21:54:59.461614 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.461588    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"]
Oct 30 21:54:59.465584 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:54:59.465561    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb47f460_f5f8_4f9f_908b_4c4532072b3f.slice/crio-bd8e11746a3de1119599d53d3b5a8c0499f095151bfc260565d57398dfd5648b WatchSource:0}: Error finding container bd8e11746a3de1119599d53d3b5a8c0499f095151bfc260565d57398dfd5648b: Status 404 returned error can't find the container with id bd8e11746a3de1119599d53d3b5a8c0499f095151bfc260565d57398dfd5648b
Oct 30 21:54:59.468279 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.468254    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d158a3d2-ee91-4ff9-8def-629768a484d2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72\" (UID: \"d158a3d2-ee91-4ff9-8def-629768a484d2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"
Oct 30 21:54:59.468451 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.468392    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjn8\" (UniqueName: \"kubernetes.io/projected/d158a3d2-ee91-4ff9-8def-629768a484d2-kube-api-access-7kjn8\") pod \"payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72\" (UID: \"d158a3d2-ee91-4ff9-8def-629768a484d2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"
Oct 30 21:54:59.469440 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.469417    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d158a3d2-ee91-4ff9-8def-629768a484d2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72\" (UID: \"d158a3d2-ee91-4ff9-8def-629768a484d2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"
Oct 30 21:54:59.530686 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.530650    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjn8\" (UniqueName: \"kubernetes.io/projected/d158a3d2-ee91-4ff9-8def-629768a484d2-kube-api-access-7kjn8\") pod \"payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72\" (UID: \"d158a3d2-ee91-4ff9-8def-629768a484d2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"
Oct 30 21:54:59.608125 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.608093    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"
Oct 30 21:54:59.834307 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.834278    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"]
Oct 30 21:54:59.835126 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:54:59.835097    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0e9ade_f4a6_4795_bda4_63df0eedf974.slice/crio-44de5bf11ad2589d64a13c4a5a7f874c19561f04b1d991dee325f18ab89345ce WatchSource:0}: Error finding container 44de5bf11ad2589d64a13c4a5a7f874c19561f04b1d991dee325f18ab89345ce: Status 404 returned error can't find the container with id 44de5bf11ad2589d64a13c4a5a7f874c19561f04b1d991dee325f18ab89345ce
Oct 30 21:54:59.943591 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.943551    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84" event={"ID":"fb47f460-f5f8-4f9f-908b-4c4532072b3f","Type":"ContainerStarted","Data":"bd8e11746a3de1119599d53d3b5a8c0499f095151bfc260565d57398dfd5648b"}
Oct 30 21:54:59.945324 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.945289    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w" event={"ID":"8f0e9ade-f4a6-4795-bda4-63df0eedf974","Type":"ContainerStarted","Data":"46333c4dbc3a2841dab5ad6398483d4c93ec962b7f83979d007ec7f765b8027e"}
Oct 30 21:54:59.945681 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.945442    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"
Oct 30 21:54:59.945681 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.945481    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w" event={"ID":"8f0e9ade-f4a6-4795-bda4-63df0eedf974","Type":"ContainerStarted","Data":"44de5bf11ad2589d64a13c4a5a7f874c19561f04b1d991dee325f18ab89345ce"}
Oct 30 21:54:59.988455 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:54:59.988417    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"]
Oct 30 21:54:59.990295 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:54:59.990273    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd158a3d2_ee91_4ff9_8def_629768a484d2.slice/crio-d75c26a22e48d0eae354bc9673b957eab159ff2e134a9784b6bafd3370450f4a WatchSource:0}: Error finding container d75c26a22e48d0eae354bc9673b957eab159ff2e134a9784b6bafd3370450f4a: Status 404 returned error can't find the container with id d75c26a22e48d0eae354bc9673b957eab159ff2e134a9784b6bafd3370450f4a
Oct 30 21:55:00.058807 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:00.058783    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:55:00.063073 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:00.063022    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w" podStartSLOduration=1.063008118 podStartE2EDuration="1.063008118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:00.06133887 +0000 UTC m=+750.795701898" watchObservedRunningTime="2026-10-30 21:55:00.063008118 +0000 UTC m=+750.797371191"
Oct 30 21:55:00.951760 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:00.951713    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72" event={"ID":"d158a3d2-ee91-4ff9-8def-629768a484d2","Type":"ContainerStarted","Data":"ad8dc129821cf2f608f82ca129dc467521d04e64f1b57ca493b2ccd1e80dd8c0"}
Oct 30 21:55:00.952002 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:00.951801    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72" event={"ID":"d158a3d2-ee91-4ff9-8def-629768a484d2","Type":"ContainerStarted","Data":"d75c26a22e48d0eae354bc9673b957eab159ff2e134a9784b6bafd3370450f4a"}
Oct 30 21:55:00.952002 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:00.951839    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"
Oct 30 21:55:00.953340 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:00.953317    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84" event={"ID":"fb47f460-f5f8-4f9f-908b-4c4532072b3f","Type":"ContainerStarted","Data":"e69b643daad0a79160f620d19e185a9124409e57f6fbc1b24bc999e49c85200e"}
Oct 30 21:55:00.953471 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:00.953457    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:55:00.953645 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:00.953629    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:55:00.954720 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:00.954700    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:55:01.067961 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:01.067903    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72" podStartSLOduration=2.067889821 podStartE2EDuration="2.067889821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:01.066173068 +0000 UTC m=+751.800536100" watchObservedRunningTime="2026-10-30 21:55:01.067889821 +0000 UTC m=+751.802252849"
Oct 30 21:55:01.304904 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:01.304803    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84" podStartSLOduration=2.342277395 podStartE2EDuration="3.304788553s" totalImagesPullingTime="962.511158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:01.30317087 +0000 UTC m=+752.037533904" watchObservedRunningTime="2026-10-30 21:55:01.304788553 +0000 UTC m=+752.039151584"
Oct 30 21:55:06.964967 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:06.964932    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 21:55:08.617405 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:08.617365    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"]
Oct 30 21:55:08.643161 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:08.642685    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:08.670952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:08.670920    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"]
Oct 30 21:55:08.733163 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:08.724966    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-b8717e-serving-cert\""
Oct 30 21:55:08.790322 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:08.790284    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sjc7\" (UniqueName: \"kubernetes.io/projected/d71f867b-5b77-4792-b2d9-71d325c58450-kube-api-access-7sjc7\") pod \"maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:08.790530 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:08.790350    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:08.790530 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:08.790479    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71f867b-5b77-4792-b2d9-71d325c58450-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:08.790644 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:08.790623    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-metrics-certs\") pod \"maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:08.891802 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:08.891694    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-metrics-certs\") pod \"maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:08.891802 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:08.891757    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sjc7\" (UniqueName: \"kubernetes.io/projected/d71f867b-5b77-4792-b2d9-71d325c58450-kube-api-access-7sjc7\") pod \"maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:08.891802 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:08.891784    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:08.892046 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:08.891806    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71f867b-5b77-4792-b2d9-71d325c58450-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:08.892046 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:55:08.891918    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-serving-cert: secret "maas-api-e2e-worker-w0-b8717e-serving-cert" not found
Oct 30 21:55:08.892046 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:55:08.891994    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-maas-api-tls podName:d71f867b-5b77-4792-b2d9-71d325c58450 nodeName:}" failed. No retries permitted until 2026-10-30 21:55:09.391971021 +0000 UTC m=+760.126334048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-maas-api-tls") pod "maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc" (UID: "d71f867b-5b77-4792-b2d9-71d325c58450") : secret "maas-api-e2e-worker-w0-b8717e-serving-cert" not found
Oct 30 21:55:08.892558 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:08.892481    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71f867b-5b77-4792-b2d9-71d325c58450-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:08.894419 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:08.894390    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-metrics-certs\") pod \"maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:08.957994 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:08.957966    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sjc7\" (UniqueName: \"kubernetes.io/projected/d71f867b-5b77-4792-b2d9-71d325c58450-kube-api-access-7sjc7\") pod \"maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:09.396682 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:09.396648    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:09.399246 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:09.399222    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:09.566632 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:09.566594    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:09.965956 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:09.965933    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"]
Oct 30 21:55:09.967656 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:55:09.967621    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71f867b_5b77_4792_b2d9_71d325c58450.slice/crio-90739373c75d9476fd094bd4aad3ed3982f70e6c02c1821f8be77d2220810b7a WatchSource:0}: Error finding container 90739373c75d9476fd094bd4aad3ed3982f70e6c02c1821f8be77d2220810b7a: Status 404 returned error can't find the container with id 90739373c75d9476fd094bd4aad3ed3982f70e6c02c1821f8be77d2220810b7a
Oct 30 21:55:10.005398 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:10.005360    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc" event={"ID":"d71f867b-5b77-4792-b2d9-71d325c58450","Type":"ContainerStarted","Data":"90739373c75d9476fd094bd4aad3ed3982f70e6c02c1821f8be77d2220810b7a"}
Oct 30 21:55:10.957512 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:10.957489    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"
Oct 30 21:55:11.048971 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:11.048933    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc" event={"ID":"d71f867b-5b77-4792-b2d9-71d325c58450","Type":"ContainerStarted","Data":"53e8fb18d7796d4e45dbc651b4956dc0d5ec173c74e52b8442dd300d2b7339ba"}
Oct 30 21:55:11.060198 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:11.060128    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:11.202814 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:11.202690    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc" podStartSLOduration=2.276790712 podStartE2EDuration="3.202666493s" totalImagesPullingTime="925.875781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:11.201888311 +0000 UTC m=+761.936251344" watchObservedRunningTime="2026-10-30 21:55:11.202666493 +0000 UTC m=+761.937029525"
Oct 30 21:55:11.962250 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:11.962222    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"
Oct 30 21:55:17.060257 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:17.060228    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 21:55:17.086566 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:17.086535    4553 generic.go:356] "Generic (PLEG): container finished" podID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" containerID="5a94a89a74ee45e53e6272d40cfd852cd87b1b8ec1fd3adf065201de71690ac9" exitCode=0
Oct 30 21:55:17.086742 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:17.086616    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv" event={"ID":"f8f35f60-8e19-4353-85eb-73b8f7a36c11","Type":"ContainerDied","Data":"5a94a89a74ee45e53e6272d40cfd852cd87b1b8ec1fd3adf065201de71690ac9"}
Oct 30 21:55:17.087015 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:17.087002    4553 scope.go:122] "RemoveContainer" containerID="5a94a89a74ee45e53e6272d40cfd852cd87b1b8ec1fd3adf065201de71690ac9"
Oct 30 21:55:17.088349 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:17.088326    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:55:17.088406 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:17.088394    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:55:17.088448 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:17.088430    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:55:18.093976 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:18.093943    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv" event={"ID":"f8f35f60-8e19-4353-85eb-73b8f7a36c11","Type":"ContainerStarted","Data":"987a0e548c22ea32592598cc7f39bb5588da95c4eb0f4028c33025969e10592d"}
Oct 30 21:55:18.459732 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:18.459630    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:55:18.459922 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:18.459765    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:55:18.459922 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:18.459809    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv" podUID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 21:55:19.459562 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:19.459524    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:55:19.460117 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:19.459572    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv" podUID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 21:55:20.459877 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:20.459832    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:55:20.460304 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:20.459898    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv" podUID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 21:55:21.459804 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:21.459772    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:55:21.459966 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:21.459844    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv" podUID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 21:55:22.459597 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:22.459567    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Oct 30 21:55:22.459756 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:22.459651    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv" podUID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Oct 30 21:55:23.468873 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:23.468838    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:55:23.469367 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:23.469164    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:55:23.470013 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:23.469995    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 21:55:25.156644 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:25.156606    4553 generic.go:356] "Generic (PLEG): container finished" podID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" containerID="57b9223804e5c881c6c852272e96b288fd26968f286e3cf5e41f0687e6444dd7" exitCode=0
Oct 30 21:55:25.157205 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:25.156684    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b" event={"ID":"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3","Type":"ContainerDied","Data":"57b9223804e5c881c6c852272e96b288fd26968f286e3cf5e41f0687e6444dd7"}
Oct 30 21:55:25.157205 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:25.157120    4553 scope.go:122] "RemoveContainer" containerID="57b9223804e5c881c6c852272e96b288fd26968f286e3cf5e41f0687e6444dd7"
Oct 30 21:55:25.158503 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:25.158478    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:55:25.158558 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:25.158547    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:55:25.158596 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:25.158580    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:55:26.165365 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:26.165325    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b" event={"ID":"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3","Type":"ContainerStarted","Data":"859ba0b5db24ba75490f7df78c49586274caf8f859d10d5c8d027b31d0bb8619"}
Oct 30 21:55:26.413158 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:26.412474    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:55:26.413158 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:26.412591    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:55:26.413158 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:26.412646    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b" podUID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 21:55:27.415572 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:27.415537    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:55:27.416217 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:27.416184    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b" podUID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 21:55:28.200324 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:55:28.200289    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:55:28.413547 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:28.413513    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:55:28.413858 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:28.413831    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b" podUID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 21:55:29.322459 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.313896    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"]
Oct 30 21:55:29.323357 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.323319    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b" podUID="04070c89-d0c2-4f45-8a25-e7a3944e3dfb" containerName="maas-api" containerID="cri-o://7c296b6dae73027b9171c39697c2c461678b49a4abb2582c732cc6952acd872f" gracePeriod=30
Oct 30 21:55:29.422166 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.413569    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:55:29.449027 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.448990    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b" podUID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 21:55:29.451580 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:55:29.451551    4553 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04070c89_d0c2_4f45_8a25_e7a3944e3dfb.slice/crio-conmon-7c296b6dae73027b9171c39697c2c461678b49a4abb2582c732cc6952acd872f.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:55:29.659228 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.659165    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:55:29.755036 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.755005    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-maas-api-tls\") pod \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") "
Oct 30 21:55:29.755271 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.755051    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-openshift-service-ca\") pod \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") "
Oct 30 21:55:29.755271 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.755115    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-kube-api-access-ncwnd\" (UniqueName: \"kubernetes.io/projected/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-kube-api-access-ncwnd\") pod \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") "
Oct 30 21:55:29.755271 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.755229    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-metrics-certs\") pod \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\" (UID: \"04070c89-d0c2-4f45-8a25-e7a3944e3dfb\") "
Oct 30 21:55:29.755485 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.755456    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-openshift-service-ca" pod "04070c89-d0c2-4f45-8a25-e7a3944e3dfb" (UID: "04070c89-d0c2-4f45-8a25-e7a3944e3dfb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:29.755652 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.755633    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:29.757455 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.757429    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-kube-api-access-ncwnd" pod "04070c89-d0c2-4f45-8a25-e7a3944e3dfb" (UID: "04070c89-d0c2-4f45-8a25-e7a3944e3dfb"). InnerVolumeSpecName "kube-api-access-ncwnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:29.757901 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.757881    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-metrics-certs" pod "04070c89-d0c2-4f45-8a25-e7a3944e3dfb" (UID: "04070c89-d0c2-4f45-8a25-e7a3944e3dfb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:29.757973 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.757906    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-maas-api-tls" pod "04070c89-d0c2-4f45-8a25-e7a3944e3dfb" (UID: "04070c89-d0c2-4f45-8a25-e7a3944e3dfb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:29.856956 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.856863    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:29.856956 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.856907    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncwnd\" (UniqueName: \"kubernetes.io/projected/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-kube-api-access-ncwnd\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:29.856956 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:29.856923    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04070c89-d0c2-4f45-8a25-e7a3944e3dfb-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:30.188922 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:30.188827    4553 generic.go:356] "Generic (PLEG): container finished" podID="04070c89-d0c2-4f45-8a25-e7a3944e3dfb" containerID="7c296b6dae73027b9171c39697c2c461678b49a4abb2582c732cc6952acd872f" exitCode=0
Oct 30 21:55:30.188922 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:30.188875    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"
Oct 30 21:55:30.189193 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:30.188921    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b" event={"ID":"04070c89-d0c2-4f45-8a25-e7a3944e3dfb","Type":"ContainerDied","Data":"7c296b6dae73027b9171c39697c2c461678b49a4abb2582c732cc6952acd872f"}
Oct 30 21:55:30.189193 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:30.188981    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b" event={"ID":"04070c89-d0c2-4f45-8a25-e7a3944e3dfb","Type":"ContainerDied","Data":"d4c7c48acfb7f2d4fa3eb11e216dd8b627534dafe7b57d1f9c41919fcac23794"}
Oct 30 21:55:30.189193 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:30.189022    4553 scope.go:122] "RemoveContainer" containerID="7c296b6dae73027b9171c39697c2c461678b49a4abb2582c732cc6952acd872f"
Oct 30 21:55:30.198920 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:30.198900    4553 scope.go:122] "RemoveContainer" containerID="7c296b6dae73027b9171c39697c2c461678b49a4abb2582c732cc6952acd872f"
Oct 30 21:55:30.199273 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:55:30.199248    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c296b6dae73027b9171c39697c2c461678b49a4abb2582c732cc6952acd872f\": container with ID starting with 7c296b6dae73027b9171c39697c2c461678b49a4abb2582c732cc6952acd872f not found: ID does not exist" containerID="7c296b6dae73027b9171c39697c2c461678b49a4abb2582c732cc6952acd872f"
Oct 30 21:55:30.199372 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:30.199280    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c296b6dae73027b9171c39697c2c461678b49a4abb2582c732cc6952acd872f"} err="failed to get container status \"7c296b6dae73027b9171c39697c2c461678b49a4abb2582c732cc6952acd872f\": rpc error: code = NotFound desc = could not find container \"7c296b6dae73027b9171c39697c2c461678b49a4abb2582c732cc6952acd872f\": container with ID starting with 7c296b6dae73027b9171c39697c2c461678b49a4abb2582c732cc6952acd872f not found: ID does not exist"
Oct 30 21:55:30.355100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:30.355065    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"]
Oct 30 21:55:30.359339 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:30.359313    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c678f1ee-656c49d8f4-cnn5b"]
Oct 30 21:55:30.417159 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:30.412566    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 21:55:30.417159 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:30.412638    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b" podUID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 21:55:31.123837 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.123801    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"]
Oct 30 21:55:31.124614 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.124594    4553 cpu_manager.go:414] "containerMap: removing container" podUID="04070c89-d0c2-4f45-8a25-e7a3944e3dfb" containerName="maas-api"
Oct 30 21:55:31.124673 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.124619    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04070c89-d0c2-4f45-8a25-e7a3944e3dfb" containerName="maas-api"
Oct 30 21:55:31.124780 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.124769    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="04070c89-d0c2-4f45-8a25-e7a3944e3dfb" containerName="maas-api"
Oct 30 21:55:31.129693 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.129670    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:31.172803 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.172769    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e0da6e-0907-469d-8d47-ff4eeb9623ac-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:31.172976 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.172902    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs54d\" (UniqueName: \"kubernetes.io/projected/62e0da6e-0907-469d-8d47-ff4eeb9623ac-kube-api-access-vs54d\") pod \"maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:31.172976 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.172962    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-metrics-certs\") pod \"maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:31.173056 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.173038    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:31.185928 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.185890    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"]
Oct 30 21:55:31.232298 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.232258    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-d2a3e0-serving-cert\""
Oct 30 21:55:31.274817 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.274774    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e0da6e-0907-469d-8d47-ff4eeb9623ac-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:31.275023 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.274946    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs54d\" (UniqueName: \"kubernetes.io/projected/62e0da6e-0907-469d-8d47-ff4eeb9623ac-kube-api-access-vs54d\") pod \"maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:31.275089 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.275020    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-metrics-certs\") pod \"maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:31.275089 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.275064    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:31.275324 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:55:31.275295    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-serving-cert: secret "maas-api-e2e-worker-w4-d2a3e0-serving-cert" not found
Oct 30 21:55:31.275410 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:55:31.275380    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-maas-api-tls podName:62e0da6e-0907-469d-8d47-ff4eeb9623ac nodeName:}" failed. No retries permitted until 2026-10-30 21:55:31.775358672 +0000 UTC m=+782.509721700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-maas-api-tls") pod "maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8" (UID: "62e0da6e-0907-469d-8d47-ff4eeb9623ac") : secret "maas-api-e2e-worker-w4-d2a3e0-serving-cert" not found
Oct 30 21:55:31.276391 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.276343    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e0da6e-0907-469d-8d47-ff4eeb9623ac-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:31.278275 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.278252    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-metrics-certs\") pod \"maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:31.338598 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.338555    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs54d\" (UniqueName: \"kubernetes.io/projected/62e0da6e-0907-469d-8d47-ff4eeb9623ac-kube-api-access-vs54d\") pod \"maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:31.399372 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.399290    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04070c89-d0c2-4f45-8a25-e7a3944e3dfb" path="/var/lib/kubelet/pods/04070c89-d0c2-4f45-8a25-e7a3944e3dfb/volumes"
Oct 30 21:55:31.415570 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.415533    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:55:31.415889 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.415863    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:55:31.416696 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.416677    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 21:55:31.787351 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.787255    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:31.790389 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:31.790356    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-maas-api-tls\") pod \"maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:32.046387 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:32.046301    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:32.421787 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:32.421756    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"]
Oct 30 21:55:32.423925 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:55:32.423872    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e0da6e_0907_469d_8d47_ff4eeb9623ac.slice/crio-aa7346e18e6389e7b5a9b880f996bc20b6dc3f97cf9926ac74cf6ac0e7331bc6 WatchSource:0}: Error finding container aa7346e18e6389e7b5a9b880f996bc20b6dc3f97cf9926ac74cf6ac0e7331bc6: Status 404 returned error can't find the container with id aa7346e18e6389e7b5a9b880f996bc20b6dc3f97cf9926ac74cf6ac0e7331bc6
Oct 30 21:55:33.209524 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:33.209484    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8" event={"ID":"62e0da6e-0907-469d-8d47-ff4eeb9623ac","Type":"ContainerStarted","Data":"aa7346e18e6389e7b5a9b880f996bc20b6dc3f97cf9926ac74cf6ac0e7331bc6"}
Oct 30 21:55:34.216960 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:34.216922    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8" event={"ID":"62e0da6e-0907-469d-8d47-ff4eeb9623ac","Type":"ContainerStarted","Data":"dc3b64fa5dca2e8931610c916dbf9cb40fc785fcaa75b3f3a75a3247de59ce26"}
Oct 30 21:55:34.217416 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:34.217073    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:34.335926 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:34.335877    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8" podStartSLOduration=2.467374399 podStartE2EDuration="3.335863773s" totalImagesPullingTime="868.489374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:34.334590949 +0000 UTC m=+785.068953980" watchObservedRunningTime="2026-10-30 21:55:34.335863773 +0000 UTC m=+785.070226852"
Oct 30 21:55:40.227648 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:40.227617    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 21:55:53.319306 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:53.319214    4553 generic.go:356] "Generic (PLEG): container finished" podID="5c189afe-afe8-4736-8e80-e321da9d7268" containerID="5db73e8a9d255a13bd7b722d14e7a376efd66a26fdf799304296c1b2f88040d0" exitCode=0
Oct 30 21:55:53.319832 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:53.319294    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q" event={"ID":"5c189afe-afe8-4736-8e80-e321da9d7268","Type":"ContainerDied","Data":"5db73e8a9d255a13bd7b722d14e7a376efd66a26fdf799304296c1b2f88040d0"}
Oct 30 21:55:53.319832 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:53.319783    4553 scope.go:122] "RemoveContainer" containerID="5db73e8a9d255a13bd7b722d14e7a376efd66a26fdf799304296c1b2f88040d0"
Oct 30 21:55:53.321176 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:53.321121    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:55:53.321253 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:53.321220    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:55:53.321292 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:53.321256    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:55:54.336798 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:54.336754    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q" event={"ID":"5c189afe-afe8-4736-8e80-e321da9d7268","Type":"ContainerStarted","Data":"3dff47f256c389f1992996fbfa0a9310e9ee0dd7d475744aedd6411c3fa71151"}
Oct 30 21:55:55.054784 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:55.054744    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:55:55.054977 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:55.054938    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:55:55.055020 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:55.054979    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q" podUID="5c189afe-afe8-4736-8e80-e321da9d7268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 21:55:56.055757 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:56.055715    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:55:56.056322 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:56.055777    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q" podUID="5c189afe-afe8-4736-8e80-e321da9d7268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 21:55:57.055043 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:57.055005    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:55:57.055241 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:57.055070    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q" podUID="5c189afe-afe8-4736-8e80-e321da9d7268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 21:55:58.054954 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:58.054920    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:55:58.055444 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:58.054976    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q" podUID="5c189afe-afe8-4736-8e80-e321da9d7268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 21:55:58.202049 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:55:58.202009    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:55:59.055321 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:59.055284    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 21:55:59.055791 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:55:59.055346    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q" podUID="5c189afe-afe8-4736-8e80-e321da9d7268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 21:56:00.059873 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:00.059840    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:56:00.060344 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:00.060169    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:56:00.060813 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:00.060799    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 21:56:28.201675 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:56:28.201636    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:56:52.762377 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:52.762345    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"]
Oct 30 21:56:52.766341 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:52.766322    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:52.871413 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:52.871388    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b8717e\"/\"openshift-service-ca.crt\""
Oct 30 21:56:52.871608 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:52.871432    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b8717e\"/\"kube-root-ca.crt\""
Oct 30 21:56:52.871608 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:52.871458    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b8717e\"/\"premium-w0-b8717e-kserve-self-signed-certs\""
Oct 30 21:56:52.871608 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:52.871522    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b8717e\"/\"default-dockercfg-6s4ht\""
Oct 30 21:56:52.873844 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:52.873827    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"]
Oct 30 21:56:52.905473 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:52.905427    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd7a0ddb-718a-41f3-8fbd-e83896d80587-tls-certs\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:52.905670 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:52.905604    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-home\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:52.905670 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:52.905649    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj8px\" (UniqueName: \"kubernetes.io/projected/fd7a0ddb-718a-41f3-8fbd-e83896d80587-kube-api-access-lj8px\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:52.905804 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:52.905704    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-dshm\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:52.905804 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:52.905753    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-tmp-dir\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:52.905804 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:52.905788    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-model-cache\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:53.007391 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.007357    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-home\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:53.007573 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.007401    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj8px\" (UniqueName: \"kubernetes.io/projected/fd7a0ddb-718a-41f3-8fbd-e83896d80587-kube-api-access-lj8px\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:53.007573 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.007432    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-dshm\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:53.007573 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.007460    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-tmp-dir\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:53.007573 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.007481    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-model-cache\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:53.007573 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.007508    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd7a0ddb-718a-41f3-8fbd-e83896d80587-tls-certs\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:53.007854 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.007831    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-home\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:53.007891 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.007843    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-tmp-dir\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:53.007949 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.007927    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-model-cache\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:53.009974 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.009957    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-dshm\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:53.010308 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.010289    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd7a0ddb-718a-41f3-8fbd-e83896d80587-tls-certs\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:53.068842 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.068764    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj8px\" (UniqueName: \"kubernetes.io/projected/fd7a0ddb-718a-41f3-8fbd-e83896d80587-kube-api-access-lj8px\") pod \"premium-w0-b8717e-kserve-dd67f47b8-kmqr4\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") " pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:53.078512 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.078490    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:53.457207 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.457177    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"]
Oct 30 21:56:53.459852 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:56:53.459824    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd7a0ddb_718a_41f3_8fbd_e83896d80587.slice/crio-60479501c262f3de93940d9f403797833a770d89163094ed11fd9bcbb13f486d WatchSource:0}: Error finding container 60479501c262f3de93940d9f403797833a770d89163094ed11fd9bcbb13f486d: Status 404 returned error can't find the container with id 60479501c262f3de93940d9f403797833a770d89163094ed11fd9bcbb13f486d
Oct 30 21:56:53.649636 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.649599    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" event={"ID":"fd7a0ddb-718a-41f3-8fbd-e83896d80587","Type":"ContainerStarted","Data":"d7df96d81c4953fb894ea174428339fd378306fae3c9c5ddc44d77b04f73ee8c"}
Oct 30 21:56:53.649808 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.649652    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" event={"ID":"fd7a0ddb-718a-41f3-8fbd-e83896d80587","Type":"ContainerStarted","Data":"60479501c262f3de93940d9f403797833a770d89163094ed11fd9bcbb13f486d"}
Oct 30 21:56:53.770355 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:53.770209    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podStartSLOduration=1.77019244 podStartE2EDuration="1.77019244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:53.769529925 +0000 UTC m=+864.503892959" watchObservedRunningTime="2026-10-30 21:56:53.77019244 +0000 UTC m=+864.504555473"
Oct 30 21:56:54.079619 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:56:54.079535    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:56:58.200251 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:56:58.200213    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:57:03.079382 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:03.079341    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:57:03.092535 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:03.092508    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:57:03.713206 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:03.713179    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 21:57:28.218073 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:57:28.218035    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:57:29.496627 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:29.496594    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 21:57:29.508030 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:29.508003    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 21:57:40.599804 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.599767    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"]
Oct 30 21:57:40.603512 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.603490    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.707720 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.707684    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1c3eef\"/\"openshift-service-ca.crt\""
Oct 30 21:57:40.707720 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.707728    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1c3eef\"/\"kube-root-ca.crt\""
Oct 30 21:57:40.707991 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.707788    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1c3eef\"/\"default-dockercfg-dc4hx\""
Oct 30 21:57:40.708628 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.708608    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"]
Oct 30 21:57:40.710527 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.710510    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1c3eef\"/\"simulator-w1-1c3eef-kserve-self-signed-certs\""
Oct 30 21:57:40.732039 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.732006    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-home\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.732446 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.732420    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-tmp-dir\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.732795 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.732731    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-model-cache\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.733156 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.733088    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ecfcbe-143b-46ef-ac5f-fed921354ef6-tls-certs\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.733438 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.733369    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-dshm\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.733720 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.733685    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phsqq\" (UniqueName: \"kubernetes.io/projected/86ecfcbe-143b-46ef-ac5f-fed921354ef6-kube-api-access-phsqq\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.835416 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.835384    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phsqq\" (UniqueName: \"kubernetes.io/projected/86ecfcbe-143b-46ef-ac5f-fed921354ef6-kube-api-access-phsqq\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.835604 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.835442    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-home\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.835604 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.835516    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-tmp-dir\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.835604 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.835577    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-model-cache\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.835765 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.835712    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ecfcbe-143b-46ef-ac5f-fed921354ef6-tls-certs\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.835765 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.835746    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-dshm\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.836032 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.836002    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-tmp-dir\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.836207 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.836074    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-home\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.836207 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.836172    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-model-cache\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.838077 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.838053    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-dshm\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.838368 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.838352    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ecfcbe-143b-46ef-ac5f-fed921354ef6-tls-certs\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.893502 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.893430    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phsqq\" (UniqueName: \"kubernetes.io/projected/86ecfcbe-143b-46ef-ac5f-fed921354ef6-kube-api-access-phsqq\") pod \"simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") " pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:40.915258 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:40.915227    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:41.291840 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:41.291806    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"]
Oct 30 21:57:41.293442 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:57:41.293409    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86ecfcbe_143b_46ef_ac5f_fed921354ef6.slice/crio-24df1a8a7c0d1fc09e09058908577ef04e297b15878152610ea281ce23ef4ebf WatchSource:0}: Error finding container 24df1a8a7c0d1fc09e09058908577ef04e297b15878152610ea281ce23ef4ebf: Status 404 returned error can't find the container with id 24df1a8a7c0d1fc09e09058908577ef04e297b15878152610ea281ce23ef4ebf
Oct 30 21:57:41.918075 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:41.918040    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" event={"ID":"86ecfcbe-143b-46ef-ac5f-fed921354ef6","Type":"ContainerStarted","Data":"62fc6fe60fc088cb27ee0b57043ac310bd52cdc63860dd1d144c076c9c505d53"}
Oct 30 21:57:41.918512 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:41.918086    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" event={"ID":"86ecfcbe-143b-46ef-ac5f-fed921354ef6","Type":"ContainerStarted","Data":"24df1a8a7c0d1fc09e09058908577ef04e297b15878152610ea281ce23ef4ebf"}
Oct 30 21:57:42.037053 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:42.036993    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podStartSLOduration=2.036978886 podStartE2EDuration="2.036978886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:42.034815391 +0000 UTC m=+912.769178422" watchObservedRunningTime="2026-10-30 21:57:42.036978886 +0000 UTC m=+912.771341916"
Oct 30 21:57:42.915563 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:42.915538    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:50.915772 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:50.915726    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:50.929089 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:50.929063    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:50.975648 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:57:50.975615    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 21:57:58.200233 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:57:58.200189    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:58:15.378073 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.377990    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"]
Oct 30 21:58:15.381957 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.381935    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.413636 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.413590    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.413799 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.413650    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.413799 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.413670    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.413799 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.413770    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.413923 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.413800    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.413923 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.413817    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.413923 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.413840    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbqmh\" (UniqueName: \"kubernetes.io/projected/8e3d7161-a39d-4501-a8e5-6f547c91237b-kube-api-access-sbqmh\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.413923 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.413906    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.414081 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.413931    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3d7161-a39d-4501-a8e5-6f547c91237b-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.434897 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.434868    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-90e0d5-openshift-default-dockercfg-mfg6n\""
Oct 30 21:58:15.488062 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.488027    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"]
Oct 30 21:58:15.514923 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.514886    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.515111 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.514930    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.515111 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.515058    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.515299 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.515160    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.515299 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.515220    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.515299 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.515251    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.515466 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.515299    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqmh\" (UniqueName: \"kubernetes.io/projected/8e3d7161-a39d-4501-a8e5-6f547c91237b-kube-api-access-sbqmh\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.515466 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.515317    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.515466 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.515354    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.515466 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.515389    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3d7161-a39d-4501-a8e5-6f547c91237b-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.515715 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.515499    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.515715 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.515537    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.515896 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.515873    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.516100 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.516080    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3d7161-a39d-4501-a8e5-6f547c91237b-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.517655 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.517633    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.517890 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.517874    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.573429 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.573401    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.573952 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.573934    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqmh\" (UniqueName: \"kubernetes.io/projected/8e3d7161-a39d-4501-a8e5-6f547c91237b-kube-api-access-sbqmh\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:15.696657 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:15.696561    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:16.128226 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:16.128198    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"]
Oct 30 21:58:16.129868 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:58:16.129845    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3d7161_a39d_4501_a8e5_6f547c91237b.slice/crio-062607f81e27b9d2238ad31ceeeb97c0ac19e0a162677542858e258fde77da18 WatchSource:0}: Error finding container 062607f81e27b9d2238ad31ceeeb97c0ac19e0a162677542858e258fde77da18: Status 404 returned error can't find the container with id 062607f81e27b9d2238ad31ceeeb97c0ac19e0a162677542858e258fde77da18
Oct 30 21:58:16.132219 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:16.132186    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:58:16.132315 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:16.132250    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:58:16.132315 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:16.132278    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:58:17.103986 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:17.103945    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk" event={"ID":"8e3d7161-a39d-4501-a8e5-6f547c91237b","Type":"ContainerStarted","Data":"d6460913d5290a32170d6c7cf8466f146a1b08b4c282020d7d15d6b496dc4bbb"}
Oct 30 21:58:17.104421 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:17.104046    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk" event={"ID":"8e3d7161-a39d-4501-a8e5-6f547c91237b","Type":"ContainerStarted","Data":"062607f81e27b9d2238ad31ceeeb97c0ac19e0a162677542858e258fde77da18"}
Oct 30 21:58:17.275333 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:17.275276    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk" podStartSLOduration=2.275261134 podStartE2EDuration="2.275261134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:17.273563554 +0000 UTC m=+948.007926607" watchObservedRunningTime="2026-10-30 21:58:17.275261134 +0000 UTC m=+948.009624165"
Oct 30 21:58:17.697303 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:17.697266    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:17.698736 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:17.698710    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:58:17.698845 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:17.698774    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk" podUID="8e3d7161-a39d-4501-a8e5-6f547c91237b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 21:58:18.697272 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:18.697238    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:58:18.697674 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:18.697308    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk" podUID="8e3d7161-a39d-4501-a8e5-6f547c91237b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 21:58:19.697968 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:19.697928    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:58:19.698381 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:19.697992    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk" podUID="8e3d7161-a39d-4501-a8e5-6f547c91237b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 21:58:20.697171 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:20.697120    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:58:20.697361 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:20.697192    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk" podUID="8e3d7161-a39d-4501-a8e5-6f547c91237b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 21:58:21.697901 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:21.697865    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:58:21.698324 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:21.697917    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk" podUID="8e3d7161-a39d-4501-a8e5-6f547c91237b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 21:58:22.951414 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:22.951381    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:22.952867 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:22.952843    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:22.954069 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:58:22.954050    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 21:58:28.202116 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:58:28.202076    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:58:58.201377 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:58:58.201334    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:59:25.525513 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.525474    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"]
Oct 30 21:59:25.529630 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.529606    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.636708 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.636665    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"]
Oct 30 21:59:25.657589 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.657557    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.657761 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.657612    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l649t\" (UniqueName: \"kubernetes.io/projected/d66e97b1-d8b1-477a-a8f7-82466dbcd432-kube-api-access-l649t\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.657761 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.657695    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.657761 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.657718    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.657761 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.657740    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.657975 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.657936    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.658031 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.657998    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.658083 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.658060    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.658179 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.658126    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.759093 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.759060    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.759335 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.759117    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.759335 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.759172    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.759335 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.759201    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l649t\" (UniqueName: \"kubernetes.io/projected/d66e97b1-d8b1-477a-a8f7-82466dbcd432-kube-api-access-l649t\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.759335 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.759226    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.759335 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.759247    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.759335 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.759277    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.759661 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.759362    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.759661 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.759400    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.759661 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.759601    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.759661 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.759645    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.760038 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.760006    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.760352 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.760106    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.760352 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.760276    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.762053 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.762031    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.762347 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.762328    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.817985 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.817918    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l649t\" (UniqueName: \"kubernetes.io/projected/d66e97b1-d8b1-477a-a8f7-82466dbcd432-kube-api-access-l649t\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.818772 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.818749    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-tdqzj\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:25.843128 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:25.843097    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:26.273492 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:26.273463    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"]
Oct 30 21:59:26.275457 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:59:26.275421    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66e97b1_d8b1_477a_a8f7_82466dbcd432.slice/crio-e920a1ab33d93c3c0db23615dad43d2f2ab1f70ec3b0e44f494ecd2c97169d2a WatchSource:0}: Error finding container e920a1ab33d93c3c0db23615dad43d2f2ab1f70ec3b0e44f494ecd2c97169d2a: Status 404 returned error can't find the container with id e920a1ab33d93c3c0db23615dad43d2f2ab1f70ec3b0e44f494ecd2c97169d2a
Oct 30 21:59:26.278018 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:26.277988    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:59:26.278114 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:26.278056    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:59:26.278114 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:26.278085    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:59:26.564087 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:26.563991    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" event={"ID":"d66e97b1-d8b1-477a-a8f7-82466dbcd432","Type":"ContainerStarted","Data":"a49f50aa9f216766b38da6c11287d01bfb2b90a7750f943277d496726956cae4"}
Oct 30 21:59:26.564087 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:26.564048    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" event={"ID":"d66e97b1-d8b1-477a-a8f7-82466dbcd432","Type":"ContainerStarted","Data":"e920a1ab33d93c3c0db23615dad43d2f2ab1f70ec3b0e44f494ecd2c97169d2a"}
Oct 30 21:59:26.683014 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:26.682124    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" podStartSLOduration=1.682106037 podStartE2EDuration="1.682106037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:26.681045791 +0000 UTC m=+1017.415408823" watchObservedRunningTime="2026-10-30 21:59:26.682106037 +0000 UTC m=+1017.416469069"
Oct 30 21:59:26.843947 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:26.843918    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:27.845564 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:27.845520    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tdqzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:59:27.845994 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:27.845589    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 21:59:28.201493 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:59:28.201411    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 21:59:28.843604 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:28.843568    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tdqzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:59:28.843824 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:28.843629    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 21:59:29.837520 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:29.837490    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-86d8965798-5kppx"]
Oct 30 21:59:29.841616 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:29.841593    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 21:59:29.843751 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:29.843725    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tdqzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:59:29.843898 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:29.843786    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 21:59:29.897046 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:29.897010    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86d8965798-5kppx"]
Oct 30 21:59:29.934943 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:29.934888    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials\") pod \"payload-processing-external-model-86d8965798-5kppx\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") " pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 21:59:29.935223 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:29.935029    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-routing-overlay-volume\") pod \"payload-processing-external-model-86d8965798-5kppx\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") " pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 21:59:29.935223 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:29.935079    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-plugins-config-volume\") pod \"payload-processing-external-model-86d8965798-5kppx\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") " pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 21:59:29.935395 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:29.935260    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97b7s\" (UniqueName: \"kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-kube-api-access-97b7s\") pod \"payload-processing-external-model-86d8965798-5kppx\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") " pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 21:59:29.944843 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:29.944810    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5j2m4\""
Oct 30 21:59:29.945013 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:29.944851    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7513640d12da\""
Oct 30 21:59:29.945013 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:29.944914    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:59:29.945013 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:29.944918    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:59:29.945013 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:29.944971    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:59:29.945400 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:29.945386    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:59:30.035840 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:30.035802    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97b7s\" (UniqueName: \"kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-kube-api-access-97b7s\") pod \"payload-processing-external-model-86d8965798-5kppx\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") " pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 21:59:30.035840 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:30.035846    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials\") pod \"payload-processing-external-model-86d8965798-5kppx\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") " pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 21:59:30.036168 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:30.035907    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-routing-overlay-volume\") pod \"payload-processing-external-model-86d8965798-5kppx\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") " pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 21:59:30.036168 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:30.035933    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-plugins-config-volume\") pod \"payload-processing-external-model-86d8965798-5kppx\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") " pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 21:59:30.036843 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:30.036816    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-routing-overlay-volume\") pod \"payload-processing-external-model-86d8965798-5kppx\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") " pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 21:59:30.036843 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:30.036834    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-plugins-config-volume\") pod \"payload-processing-external-model-86d8965798-5kppx\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") " pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 21:59:30.038619 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:30.038592    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials\") pod \"payload-processing-external-model-86d8965798-5kppx\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") " pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 21:59:30.093788 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:30.093713    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97b7s\" (UniqueName: \"kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-kube-api-access-97b7s\") pod \"payload-processing-external-model-86d8965798-5kppx\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") " pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 21:59:30.156820 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:30.156784    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 21:59:30.558571 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:30.558545    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86d8965798-5kppx"]
Oct 30 21:59:30.559963 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:59:30.559934    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1aa9e6_3402_4795_b4f0_33c16d6856f0.slice/crio-950c50d437ad71fc11a1f0c7e152399aabeaa8236e04c65cf0386a7d2b87a7a5 WatchSource:0}: Error finding container 950c50d437ad71fc11a1f0c7e152399aabeaa8236e04c65cf0386a7d2b87a7a5: Status 404 returned error can't find the container with id 950c50d437ad71fc11a1f0c7e152399aabeaa8236e04c65cf0386a7d2b87a7a5
Oct 30 21:59:30.591791 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:30.591762    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" event={"ID":"ab1aa9e6-3402-4795-b4f0-33c16d6856f0","Type":"ContainerStarted","Data":"950c50d437ad71fc11a1f0c7e152399aabeaa8236e04c65cf0386a7d2b87a7a5"}
Oct 30 21:59:30.844436 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:30.844408    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tdqzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:59:30.844942 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:30.844462    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 21:59:31.598803 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:31.598768    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" event={"ID":"ab1aa9e6-3402-4795-b4f0-33c16d6856f0","Type":"ContainerStarted","Data":"1a9e261d8a6a72c927e0eff4324cc2e50e05fafa1a1c33238b96418eab2e9444"}
Oct 30 21:59:31.598990 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:31.598851    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 21:59:31.843552 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:31.843516    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tdqzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:59:31.843739 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:31.843571    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 21:59:32.843708 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:32.843676    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tdqzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:59:32.844227 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:32.843734    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 21:59:33.844435 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:33.844401    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tdqzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 21:59:33.844933 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:33.844474    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 21:59:34.936865 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:34.936835    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:34.937423 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:34.937402    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:34.938273 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:34.938247    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 21:59:35.106146 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:35.106062    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" podStartSLOduration=6.106042782 podStartE2EDuration="6.106042782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:31.733979107 +0000 UTC m=+1022.468342140" watchObservedRunningTime="2026-10-30 21:59:35.106042782 +0000 UTC m=+1025.840405813"
Oct 30 21:59:40.537913 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.537876    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"]
Oct 30 21:59:40.542034 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.542006    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.598391 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.598356    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.598391 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.598400    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.598606 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.598425    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.598606 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.598519    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.598606 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.598576    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.598706 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.598635    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.598706 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.598696    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vnlv\" (UniqueName: \"kubernetes.io/projected/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-kube-api-access-7vnlv\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.598769 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.598725    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.598802 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.598784    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.643601 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.643561    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"]
Oct 30 21:59:40.699689 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.699653    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.699689 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.699694    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.699930 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.699718    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.699930 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.699893    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.700010 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.699975    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.700043 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.700027    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.700087 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.700061    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vnlv\" (UniqueName: \"kubernetes.io/projected/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-kube-api-access-7vnlv\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.700171 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.700092    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.700171 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.700093    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.700281 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.700206    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.700398 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.700364    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.700537 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.700467    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.700600 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.700538    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.700636 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.700600    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.702205 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.702188    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.702713 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.702691    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.757643 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.757613    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.757887 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.757861    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vnlv\" (UniqueName: \"kubernetes.io/projected/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-kube-api-access-7vnlv\") pod \"maas-default-gateway-openshift-default-694db55c87-krsdr\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:40.856817 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:40.856784    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:41.287453 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:41.287428    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"]
Oct 30 21:59:41.289110 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:59:41.289086    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc77e33_55c0_4f13_8ff1_3ab8706e4ec7.slice/crio-85ea78ec4d9adc178301b0da1033f2e308fde35ce36a71ad349529152032d5aa WatchSource:0}: Error finding container 85ea78ec4d9adc178301b0da1033f2e308fde35ce36a71ad349529152032d5aa: Status 404 returned error can't find the container with id 85ea78ec4d9adc178301b0da1033f2e308fde35ce36a71ad349529152032d5aa
Oct 30 21:59:41.291580 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:41.291553    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:59:41.291647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:41.291614    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:59:41.291647 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:41.291642    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:59:41.705328 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:41.705288    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" event={"ID":"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7","Type":"ContainerStarted","Data":"8e8a20b3f6e0915714ac7b35c7bfe1792148ae612743285943093523989f819a"}
Oct 30 21:59:41.705782 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:41.705357    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" event={"ID":"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7","Type":"ContainerStarted","Data":"85ea78ec4d9adc178301b0da1033f2e308fde35ce36a71ad349529152032d5aa"}
Oct 30 21:59:41.833159 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:41.831160    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" podStartSLOduration=1.83111628 podStartE2EDuration="1.83111628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:41.830492016 +0000 UTC m=+1032.564855047" watchObservedRunningTime="2026-10-30 21:59:41.83111628 +0000 UTC m=+1032.565479306"
Oct 30 21:59:41.856930 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:41.856893    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:42.625279 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:42.625234    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" podUID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:59:42.858846 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:42.858816    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-krsdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:59:42.859554 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:42.858882    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 21:59:43.857578 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:43.857545    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-krsdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:59:43.857775 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:43.857596    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 21:59:44.858697 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:44.858661    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-krsdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:59:44.859405 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:44.859373    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 21:59:45.858045 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:45.858015    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-krsdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:59:45.858242 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:45.858066    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 21:59:46.857811 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:46.857780    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-krsdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:59:46.858239 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:46.857833    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 21:59:47.857173 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:47.857114    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-krsdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:59:47.857357 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:47.857203    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 21:59:48.858148 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:48.858106    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-krsdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:59:48.858566 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:48.858205    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 21:59:49.900913 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:49.900863    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:49.901567 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:49.901230    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:49.902785 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:49.902764    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 21:59:52.606873 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:52.606831    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" podUID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 21:59:53.893070 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:53.893039    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-lxjxj-j9lmj"]
Oct 30 21:59:53.896653 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:53.896637    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 21:59:53.955381 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:53.955349    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lxjxj-j9lmj"]
Oct 30 21:59:53.964618 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:53.964582    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b16b1ca0-3042-4153-bfeb-91f6addd227b-archives-path\") pod \"periodic-gathering-lxjxj-j9lmj\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") " pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 21:59:53.964766 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:53.964678    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b16b1ca0-3042-4153-bfeb-91f6addd227b-service-ca-bundle\") pod \"periodic-gathering-lxjxj-j9lmj\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") " pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 21:59:53.964766 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:53.964704    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hm49\" (UniqueName: \"kubernetes.io/projected/b16b1ca0-3042-4153-bfeb-91f6addd227b-kube-api-access-4hm49\") pod \"periodic-gathering-lxjxj-j9lmj\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") " pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 21:59:53.964855 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:53.964815    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b16b1ca0-3042-4153-bfeb-91f6addd227b-trusted-ca-bundle\") pod \"periodic-gathering-lxjxj-j9lmj\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") " pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 21:59:54.002973 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.002942    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:59:54.066568 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.066539    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b16b1ca0-3042-4153-bfeb-91f6addd227b-service-ca-bundle\") pod \"periodic-gathering-lxjxj-j9lmj\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") " pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 21:59:54.066568 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.066574    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hm49\" (UniqueName: \"kubernetes.io/projected/b16b1ca0-3042-4153-bfeb-91f6addd227b-kube-api-access-4hm49\") pod \"periodic-gathering-lxjxj-j9lmj\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") " pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 21:59:54.066853 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.066707    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b16b1ca0-3042-4153-bfeb-91f6addd227b-trusted-ca-bundle\") pod \"periodic-gathering-lxjxj-j9lmj\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") " pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 21:59:54.066933 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.066874    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b16b1ca0-3042-4153-bfeb-91f6addd227b-archives-path\") pod \"periodic-gathering-lxjxj-j9lmj\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") " pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 21:59:54.067274 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.067253    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b16b1ca0-3042-4153-bfeb-91f6addd227b-archives-path\") pod \"periodic-gathering-lxjxj-j9lmj\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") " pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 21:59:54.067469 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.067447    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b16b1ca0-3042-4153-bfeb-91f6addd227b-service-ca-bundle\") pod \"periodic-gathering-lxjxj-j9lmj\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") " pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 21:59:54.105399 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.105371    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xlldj\""
Oct 30 21:59:54.105525 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.105481    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:59:54.108510 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.108491    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b16b1ca0-3042-4153-bfeb-91f6addd227b-trusted-ca-bundle\") pod \"periodic-gathering-lxjxj-j9lmj\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") " pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 21:59:54.157869 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.157809    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hm49\" (UniqueName: \"kubernetes.io/projected/b16b1ca0-3042-4153-bfeb-91f6addd227b-kube-api-access-4hm49\") pod \"periodic-gathering-lxjxj-j9lmj\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") " pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 21:59:54.211519 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.211489    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 21:59:54.592247 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.590506    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lxjxj-j9lmj"]
Oct 30 21:59:54.595673 ip-10-0-133-226 kubenswrapper[4553]: W1030 21:59:54.595639    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16b1ca0_3042_4153_bfeb_91f6addd227b.slice/crio-86666cdc9c28632c06e953328dbb8449dc239959acd16aff854a20c1fa8f8f73 WatchSource:0}: Error finding container 86666cdc9c28632c06e953328dbb8449dc239959acd16aff854a20c1fa8f8f73: Status 404 returned error can't find the container with id 86666cdc9c28632c06e953328dbb8449dc239959acd16aff854a20c1fa8f8f73
Oct 30 21:59:54.812472 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.812439    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lxjxj-j9lmj" event={"ID":"b16b1ca0-3042-4153-bfeb-91f6addd227b","Type":"ContainerStarted","Data":"b0eeb9dd787d7e253f64b50538cadd5327b2a9ebfa6fdf269d0e37f0d4f1b5fd"}
Oct 30 21:59:54.812670 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.812490    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lxjxj-j9lmj" event={"ID":"b16b1ca0-3042-4153-bfeb-91f6addd227b","Type":"ContainerStarted","Data":"86666cdc9c28632c06e953328dbb8449dc239959acd16aff854a20c1fa8f8f73"}
Oct 30 21:59:54.931807 ip-10-0-133-226 kubenswrapper[4553]: I1030 21:59:54.931665    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-lxjxj-j9lmj" podStartSLOduration=1.931643083 podStartE2EDuration="1.931643083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:54.9304213 +0000 UTC m=+1045.664784331" watchObservedRunningTime="2026-10-30 21:59:54.931643083 +0000 UTC m=+1045.666006113"
Oct 30 21:59:58.202213 ip-10-0-133-226 kubenswrapper[4553]: E1030 21:59:58.202176    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:00:00.290524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.290486    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"]
Oct 30 22:00:00.295273 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.295251    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.365152 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.365113    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1c3eef\"/\"distinct-w1-1c3eef-kserve-self-signed-certs\""
Oct 30 22:00:00.401679 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.401647    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"]
Oct 30 22:00:00.449722 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.449042    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vjwk\" (UniqueName: \"kubernetes.io/projected/d3f74f0b-774f-4b9f-ada4-165542267ef4-kube-api-access-5vjwk\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.450271 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.450240    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-home\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.450553 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.450531    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-tmp-dir\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.450830 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.450806    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f74f0b-774f-4b9f-ada4-165542267ef4-tls-certs\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.451031 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.451014    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-model-cache\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.451281 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.451264    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-dshm\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.552640 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.552539    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-home\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.552640 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.552612    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-tmp-dir\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.552886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.552687    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f74f0b-774f-4b9f-ada4-165542267ef4-tls-certs\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.552886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.552714    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-model-cache\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.552886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.552743    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-dshm\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.552886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.552817    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vjwk\" (UniqueName: \"kubernetes.io/projected/d3f74f0b-774f-4b9f-ada4-165542267ef4-kube-api-access-5vjwk\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.553055 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.553033    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-home\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.553098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.553062    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-tmp-dir\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.553178 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.553110    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-model-cache\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.555501 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.555472    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-dshm\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.555855 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.555835    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f74f0b-774f-4b9f-ada4-165542267ef4-tls-certs\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.611846 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.611819    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vjwk\" (UniqueName: \"kubernetes.io/projected/d3f74f0b-774f-4b9f-ada4-165542267ef4-kube-api-access-5vjwk\") pod \"distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:00.908557 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:00.908517    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:01.284689 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:01.284667    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"]
Oct 30 22:00:01.285742 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:00:01.285714    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3f74f0b_774f_4b9f_ada4_165542267ef4.slice/crio-d11cce11eb61992a081dcd841c5e13567ac4f524a661dc3350de87b76360d020 WatchSource:0}: Error finding container d11cce11eb61992a081dcd841c5e13567ac4f524a661dc3350de87b76360d020: Status 404 returned error can't find the container with id d11cce11eb61992a081dcd841c5e13567ac4f524a661dc3350de87b76360d020
Oct 30 22:00:01.854226 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:01.854182    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" event={"ID":"d3f74f0b-774f-4b9f-ada4-165542267ef4","Type":"ContainerStarted","Data":"9dc2dc024565c85f118a71439638e356e28a603bc750fbb0716f7375b88b29cb"}
Oct 30 22:00:01.854646 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:01.854243    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" event={"ID":"d3f74f0b-774f-4b9f-ada4-165542267ef4","Type":"ContainerStarted","Data":"d11cce11eb61992a081dcd841c5e13567ac4f524a661dc3350de87b76360d020"}
Oct 30 22:00:01.909443 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:01.909396    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:01.971730 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:01.971678    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podStartSLOduration=1.971665175 podStartE2EDuration="1.971665175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.970582487 +0000 UTC m=+1052.704945523" watchObservedRunningTime="2026-10-30 22:00:01.971665175 +0000 UTC m=+1052.706028205"
Oct 30 22:00:02.607503 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:02.607458    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" podUID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:00:10.908971 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:10.908936    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:10.925373 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:10.925344    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:11.919377 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:11.919350    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:00:12.607282 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:12.607251    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" podUID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:00:22.607315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:22.607267    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" podUID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:00:28.202235 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:00:28.202192    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:00:29.606602 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:29.606573    4553 scope.go:122] "RemoveContainer" containerID="6cad21f92fbe43c0f129307013f6d153e7d4413f1fd42d6d74b5be41e60edae3"
Oct 30 22:00:32.607395 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:32.607352    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" podUID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:00:42.034006 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.033972    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"]
Oct 30 22:00:42.039546 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.039525    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.064536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.064505    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-home\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.064712 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.064558    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-model-cache\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.064786 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.064734    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306879d4-4954-407f-b152-e1921381ac19-tls-certs\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.064786 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.064772    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pbp5\" (UniqueName: \"kubernetes.io/projected/306879d4-4954-407f-b152-e1921381ac19-kube-api-access-7pbp5\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.064881 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.064862    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-dshm\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.064998 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.064977    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-tmp-dir\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.095260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.095213    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"]
Oct 30 22:00:42.142364 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.142333    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b8717e\"/\"e2e-conflict-b-9b1cb2d8-kserve-self-signed-certs\""
Oct 30 22:00:42.166063 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.166024    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306879d4-4954-407f-b152-e1921381ac19-tls-certs\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.166063 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.166064    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pbp5\" (UniqueName: \"kubernetes.io/projected/306879d4-4954-407f-b152-e1921381ac19-kube-api-access-7pbp5\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.166279 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.166109    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-dshm\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.166279 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.166207    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-tmp-dir\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.166386 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.166309    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-home\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.166386 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.166368    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-model-cache\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.166695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.166671    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-tmp-dir\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.166803 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.166742    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-home\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.166803 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.166794    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-model-cache\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.203182 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.184888    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306879d4-4954-407f-b152-e1921381ac19-tls-certs\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.207175 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.188629    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-dshm\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.227702 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.227662    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pbp5\" (UniqueName: \"kubernetes.io/projected/306879d4-4954-407f-b152-e1921381ac19-kube-api-access-7pbp5\") pod \"e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") " pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.351567 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.351531    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:42.607063 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.606960    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" podUID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:00:42.775286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:42.775258    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"]
Oct 30 22:00:42.776410 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:00:42.776366    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod306879d4_4954_407f_b152_e1921381ac19.slice/crio-c64bec7973c475efb505e498fc7d5410ca2972b6487cf2d39e623fa8a3219c98 WatchSource:0}: Error finding container c64bec7973c475efb505e498fc7d5410ca2972b6487cf2d39e623fa8a3219c98: Status 404 returned error can't find the container with id c64bec7973c475efb505e498fc7d5410ca2972b6487cf2d39e623fa8a3219c98
Oct 30 22:00:43.114907 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:43.114865    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk" event={"ID":"306879d4-4954-407f-b152-e1921381ac19","Type":"ContainerStarted","Data":"90d2f9f62877017b2a62afdefd0464f3d58a8c80a0949569290df2665d10fd8c"}
Oct 30 22:00:43.115394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:43.114920    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk" event={"ID":"306879d4-4954-407f-b152-e1921381ac19","Type":"ContainerStarted","Data":"c64bec7973c475efb505e498fc7d5410ca2972b6487cf2d39e623fa8a3219c98"}
Oct 30 22:00:43.247591 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:43.247531    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk" podStartSLOduration=2.247514848 podStartE2EDuration="2.247514848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:43.245177797 +0000 UTC m=+1093.979540828" watchObservedRunningTime="2026-10-30 22:00:43.247514848 +0000 UTC m=+1093.981877882"
Oct 30 22:00:43.352477 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:43.352442    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:52.352722 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:52.352689    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:52.365741 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:52.365713    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:52.607043 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:52.606950    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" podUID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:00:53.189119 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:00:53.189088    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:00:58.201828 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:00:58.201775    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:01:01.386833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:01.386791    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" podUID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:01:01.550507 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:01.550460    4553 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7513640d12da: secret "agc-cred-7513640d12da" not found
Oct 30 22:01:01.550507 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:01.550499    4553 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-86d8965798-5kppx: secret "agc-cred-7513640d12da" not found
Oct 30 22:01:01.550743 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:01.550581    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials podName:ab1aa9e6-3402-4795-b4f0-33c16d6856f0 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:02.050558322 +0000 UTC m=+1112.784921351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials") pod "payload-processing-external-model-86d8965798-5kppx" (UID: "ab1aa9e6-3402-4795-b4f0-33c16d6856f0") : secret "agc-cred-7513640d12da" not found
Oct 30 22:01:02.057279 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:02.057238    4553 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7513640d12da: secret "agc-cred-7513640d12da" not found
Oct 30 22:01:02.057279 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:02.057275    4553 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-86d8965798-5kppx: secret "agc-cred-7513640d12da" not found
Oct 30 22:01:02.057526 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:02.057353    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials podName:ab1aa9e6-3402-4795-b4f0-33c16d6856f0 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:03.057330783 +0000 UTC m=+1113.791693809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials") pod "payload-processing-external-model-86d8965798-5kppx" (UID: "ab1aa9e6-3402-4795-b4f0-33c16d6856f0") : secret "agc-cred-7513640d12da" not found
Oct 30 22:01:03.069004 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:03.068970    4553 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7513640d12da: secret "agc-cred-7513640d12da" not found
Oct 30 22:01:03.069004 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:03.069001    4553 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-86d8965798-5kppx: secret "agc-cred-7513640d12da" not found
Oct 30 22:01:03.069497 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:03.069056    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials podName:ab1aa9e6-3402-4795-b4f0-33c16d6856f0 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:05.069042414 +0000 UTC m=+1115.803405424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials") pod "payload-processing-external-model-86d8965798-5kppx" (UID: "ab1aa9e6-3402-4795-b4f0-33c16d6856f0") : secret "agc-cred-7513640d12da" not found
Oct 30 22:01:03.608443 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:03.608407    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86d8965798-5kppx"]
Oct 30 22:01:03.609990 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:03.609957    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" podUID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:01:05.093982 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:05.093948    4553 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7513640d12da: secret "agc-cred-7513640d12da" not found
Oct 30 22:01:05.093982 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:05.093981    4553 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-86d8965798-5kppx: secret "agc-cred-7513640d12da" not found
Oct 30 22:01:05.094434 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:05.094084    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials podName:ab1aa9e6-3402-4795-b4f0-33c16d6856f0 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:09.094061185 +0000 UTC m=+1119.828424194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials") pod "payload-processing-external-model-86d8965798-5kppx" (UID: "ab1aa9e6-3402-4795-b4f0-33c16d6856f0") : secret "agc-cred-7513640d12da" not found
Oct 30 22:01:08.608960 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:08.608865    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" podUID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" containerName="payload-processing" containerID="cri-o://1a9e261d8a6a72c927e0eff4324cc2e50e05fafa1a1c33238b96418eab2e9444" gracePeriod=25
Oct 30 22:01:09.232042 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:09.230334    4553 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7513640d12da: secret "agc-cred-7513640d12da" not found
Oct 30 22:01:09.232042 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:09.230372    4553 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-86d8965798-5kppx: secret "agc-cred-7513640d12da" not found
Oct 30 22:01:09.232042 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:09.230461    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials podName:ab1aa9e6-3402-4795-b4f0-33c16d6856f0 nodeName:}" failed. No retries permitted until 2026-10-30 22:01:17.230436585 +0000 UTC m=+1127.964799610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials") pod "payload-processing-external-model-86d8965798-5kppx" (UID: "ab1aa9e6-3402-4795-b4f0-33c16d6856f0") : secret "agc-cred-7513640d12da" not found
Oct 30 22:01:09.531226 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:09.530342    4553 generic.go:356] "Generic (PLEG): container finished" podID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" containerID="1a9e261d8a6a72c927e0eff4324cc2e50e05fafa1a1c33238b96418eab2e9444" exitCode=0
Oct 30 22:01:09.531226 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:09.530513    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" event={"ID":"ab1aa9e6-3402-4795-b4f0-33c16d6856f0","Type":"ContainerDied","Data":"1a9e261d8a6a72c927e0eff4324cc2e50e05fafa1a1c33238b96418eab2e9444"}
Oct 30 22:01:09.547379 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:09.547345    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 22:01:09.668162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:09.667542    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-kube-api-access-97b7s\" (UniqueName: \"kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-kube-api-access-97b7s\") pod \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") "
Oct 30 22:01:09.668162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:09.667668    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-plugins-config-volume\") pod \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") "
Oct 30 22:01:09.668162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:09.667744    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-routing-overlay-volume\") pod \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") "
Oct 30 22:01:09.668162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:09.667820    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials\") pod \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\" (UID: \"ab1aa9e6-3402-4795-b4f0-33c16d6856f0\") "
Oct 30 22:01:09.671861 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:09.671826    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-routing-overlay-volume" pod "ab1aa9e6-3402-4795-b4f0-33c16d6856f0" (UID: "ab1aa9e6-3402-4795-b4f0-33c16d6856f0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:09.680756 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:09.680702    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-plugins-config-volume" pod "ab1aa9e6-3402-4795-b4f0-33c16d6856f0" (UID: "ab1aa9e6-3402-4795-b4f0-33c16d6856f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:09.703634 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:09.703493    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-kube-api-access-97b7s" pod "ab1aa9e6-3402-4795-b4f0-33c16d6856f0" (UID: "ab1aa9e6-3402-4795-b4f0-33c16d6856f0"). InnerVolumeSpecName "kube-api-access-97b7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:09.707758 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:09.707725    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials" pod "ab1aa9e6-3402-4795-b4f0-33c16d6856f0" (UID: "ab1aa9e6-3402-4795-b4f0-33c16d6856f0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:09.813224 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:09.781806    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97b7s\" (UniqueName: \"kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-kube-api-access-97b7s\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:01:09.813224 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:09.781846    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:01:09.813224 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:09.781881    4553 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-routing-overlay-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:01:09.813224 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:09.781897    4553 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab1aa9e6-3402-4795-b4f0-33c16d6856f0-provider-credentials\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:01:10.564120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:10.564079    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx" event={"ID":"ab1aa9e6-3402-4795-b4f0-33c16d6856f0","Type":"ContainerDied","Data":"950c50d437ad71fc11a1f0c7e152399aabeaa8236e04c65cf0386a7d2b87a7a5"}
Oct 30 22:01:10.564462 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:10.564439    4553 scope.go:122] "RemoveContainer" containerID="1a9e261d8a6a72c927e0eff4324cc2e50e05fafa1a1c33238b96418eab2e9444"
Oct 30 22:01:10.564763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:10.564742    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86d8965798-5kppx"
Oct 30 22:01:10.789651 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:10.789614    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86d8965798-5kppx"]
Oct 30 22:01:10.834262 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:10.834229    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-86d8965798-5kppx"]
Oct 30 22:01:11.393357 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:11.393323    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" path="/var/lib/kubelet/pods/ab1aa9e6-3402-4795-b4f0-33c16d6856f0/volumes"
Oct 30 22:01:28.202458 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:28.202419    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:01:30.675583 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:30.675549    4553 generic.go:356] "Generic (PLEG): container finished" podID="b16b1ca0-3042-4153-bfeb-91f6addd227b" containerID="b0eeb9dd787d7e253f64b50538cadd5327b2a9ebfa6fdf269d0e37f0d4f1b5fd" exitCode=0
Oct 30 22:01:30.676002 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:30.675623    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lxjxj-j9lmj" event={"ID":"b16b1ca0-3042-4153-bfeb-91f6addd227b","Type":"ContainerDied","Data":"b0eeb9dd787d7e253f64b50538cadd5327b2a9ebfa6fdf269d0e37f0d4f1b5fd"}
Oct 30 22:01:31.876613 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:31.876589    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 22:01:32.025843 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:32.025754    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b16b1ca0-3042-4153-bfeb-91f6addd227b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b16b1ca0-3042-4153-bfeb-91f6addd227b-archives-path\") pod \"b16b1ca0-3042-4153-bfeb-91f6addd227b\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") "
Oct 30 22:01:32.026017 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:32.025953    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b16b1ca0-3042-4153-bfeb-91f6addd227b-kube-api-access-4hm49\" (UniqueName: \"kubernetes.io/projected/b16b1ca0-3042-4153-bfeb-91f6addd227b-kube-api-access-4hm49\") pod \"b16b1ca0-3042-4153-bfeb-91f6addd227b\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") "
Oct 30 22:01:32.026085 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:32.026031    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b16b1ca0-3042-4153-bfeb-91f6addd227b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b16b1ca0-3042-4153-bfeb-91f6addd227b-trusted-ca-bundle\") pod \"b16b1ca0-3042-4153-bfeb-91f6addd227b\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") "
Oct 30 22:01:32.026085 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:32.026074    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b16b1ca0-3042-4153-bfeb-91f6addd227b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b16b1ca0-3042-4153-bfeb-91f6addd227b-service-ca-bundle\") pod \"b16b1ca0-3042-4153-bfeb-91f6addd227b\" (UID: \"b16b1ca0-3042-4153-bfeb-91f6addd227b\") "
Oct 30 22:01:32.026234 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:32.026159    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16b1ca0-3042-4153-bfeb-91f6addd227b-archives-path" pod "b16b1ca0-3042-4153-bfeb-91f6addd227b" (UID: "b16b1ca0-3042-4153-bfeb-91f6addd227b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:32.026527 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:32.026499    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16b1ca0-3042-4153-bfeb-91f6addd227b-trusted-ca-bundle" pod "b16b1ca0-3042-4153-bfeb-91f6addd227b" (UID: "b16b1ca0-3042-4153-bfeb-91f6addd227b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:32.026608 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:32.026515    4553 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b16b1ca0-3042-4153-bfeb-91f6addd227b-archives-path\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:01:32.026649 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:32.026603    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16b1ca0-3042-4153-bfeb-91f6addd227b-service-ca-bundle" pod "b16b1ca0-3042-4153-bfeb-91f6addd227b" (UID: "b16b1ca0-3042-4153-bfeb-91f6addd227b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:32.028478 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:32.028454    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16b1ca0-3042-4153-bfeb-91f6addd227b-kube-api-access-4hm49" pod "b16b1ca0-3042-4153-bfeb-91f6addd227b" (UID: "b16b1ca0-3042-4153-bfeb-91f6addd227b"). InnerVolumeSpecName "kube-api-access-4hm49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:32.127692 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:32.127653    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hm49\" (UniqueName: \"kubernetes.io/projected/b16b1ca0-3042-4153-bfeb-91f6addd227b-kube-api-access-4hm49\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:01:32.127692 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:32.127687    4553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b16b1ca0-3042-4153-bfeb-91f6addd227b-trusted-ca-bundle\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:01:32.127692 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:32.127696    4553 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b16b1ca0-3042-4153-bfeb-91f6addd227b-service-ca-bundle\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:01:32.689265 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:32.689233    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lxjxj-j9lmj"
Oct 30 22:01:32.689475 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:32.689277    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lxjxj-j9lmj" event={"ID":"b16b1ca0-3042-4153-bfeb-91f6addd227b","Type":"ContainerDied","Data":"86666cdc9c28632c06e953328dbb8449dc239959acd16aff854a20c1fa8f8f73"}
Oct 30 22:01:32.689475 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:32.689325    4553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86666cdc9c28632c06e953328dbb8449dc239959acd16aff854a20c1fa8f8f73"
Oct 30 22:01:34.227358 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.227326    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"]
Oct 30 22:01:34.227897 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.227883    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" containerName="payload-processing"
Oct 30 22:01:34.227958 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.227898    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" containerName="payload-processing"
Oct 30 22:01:34.227958 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.227910    4553 cpu_manager.go:414] "containerMap: removing container" podUID="b16b1ca0-3042-4153-bfeb-91f6addd227b" containerName="insights-gathering"
Oct 30 22:01:34.227958 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.227916    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b16b1ca0-3042-4153-bfeb-91f6addd227b" containerName="insights-gathering"
Oct 30 22:01:34.228060 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.227988    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1aa9e6-3402-4795-b4f0-33c16d6856f0" containerName="payload-processing"
Oct 30 22:01:34.228060 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.227998    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="b16b1ca0-3042-4153-bfeb-91f6addd227b" containerName="insights-gathering"
Oct 30 22:01:34.231606 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.231588    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.335113 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.335083    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1c3eef\"/\"distinct-2-w1-1c3eef-kserve-self-signed-certs\""
Oct 30 22:01:34.337509 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.337488    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"]
Oct 30 22:01:34.356387 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.356357    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-tmp-dir\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.356532 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.356403    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-model-cache\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.356532 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.356479    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4925f432-7a20-48b1-a052-d8ca2fd156e8-tls-certs\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.356615 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.356603    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-dshm\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.356701 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.356684    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-home\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.356740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.356713    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8ldh\" (UniqueName: \"kubernetes.io/projected/4925f432-7a20-48b1-a052-d8ca2fd156e8-kube-api-access-w8ldh\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.458316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.458280    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-home\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.458316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.458319    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8ldh\" (UniqueName: \"kubernetes.io/projected/4925f432-7a20-48b1-a052-d8ca2fd156e8-kube-api-access-w8ldh\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.458552 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.458393    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-tmp-dir\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.458552 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.458427    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-model-cache\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.458552 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.458450    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4925f432-7a20-48b1-a052-d8ca2fd156e8-tls-certs\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.458688 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.458571    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-dshm\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.458779 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.458731    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-home\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.458779 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.458762    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-tmp-dir\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.458984 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.458822    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-model-cache\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.461054 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.461031    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-dshm\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.461184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.461165    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4925f432-7a20-48b1-a052-d8ca2fd156e8-tls-certs\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.516053 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.515975    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8ldh\" (UniqueName: \"kubernetes.io/projected/4925f432-7a20-48b1-a052-d8ca2fd156e8-kube-api-access-w8ldh\") pod \"distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") " pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.544334 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.544303    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:34.971014 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:34.970984    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"]
Oct 30 22:01:34.972537 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:01:34.972510    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4925f432_7a20_48b1_a052_d8ca2fd156e8.slice/crio-01c71dd029c4ca925488f2264dc47e5e9392934a8b678eb77ab953795df1e26c WatchSource:0}: Error finding container 01c71dd029c4ca925488f2264dc47e5e9392934a8b678eb77ab953795df1e26c: Status 404 returned error can't find the container with id 01c71dd029c4ca925488f2264dc47e5e9392934a8b678eb77ab953795df1e26c
Oct 30 22:01:35.707704 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:35.707664    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" event={"ID":"4925f432-7a20-48b1-a052-d8ca2fd156e8","Type":"ContainerStarted","Data":"a1cb13aab1c59d7a551a4e5a46b14c85c0be3f60b1397715a50b29b7be145199"}
Oct 30 22:01:35.708110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:35.707718    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" event={"ID":"4925f432-7a20-48b1-a052-d8ca2fd156e8","Type":"ContainerStarted","Data":"01c71dd029c4ca925488f2264dc47e5e9392934a8b678eb77ab953795df1e26c"}
Oct 30 22:01:35.824704 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:35.824646    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podStartSLOduration=1.824626897 podStartE2EDuration="1.824626897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:35.823037938 +0000 UTC m=+1146.557400971" watchObservedRunningTime="2026-10-30 22:01:35.824626897 +0000 UTC m=+1146.558989930"
Oct 30 22:01:36.544827 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:36.544795    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:44.545334 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:44.545303    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:44.558316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:44.558292    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:44.798679 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:44.798603    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:01:55.684864 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:01:55.684826    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"]
Oct 30 22:01:58.202752 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:01:58.202715    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:02:09.782975 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.782946    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"]
Oct 30 22:02:09.787086 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.787062    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.818548 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.818515    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.818703 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.818553    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.818703 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.818590    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.818806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.818724    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmgzh\" (UniqueName: \"kubernetes.io/projected/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-kube-api-access-wmgzh\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.818806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.818759    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.818892 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.818865    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.818936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.818898    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.818936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.818922    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.819037 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.819020    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.889437 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.889403    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"]
Oct 30 22:02:09.919787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.919758    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.919970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.919794    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.919970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.919841    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.919970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.919883    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmgzh\" (UniqueName: \"kubernetes.io/projected/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-kube-api-access-wmgzh\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.921003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.920352    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.921294 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.921247    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.921435 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.921371    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.921763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.921708    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.922110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.921405    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.922420 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.922108    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.922551 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.922409    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.922667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.922613    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.922782 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.922759    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.922935 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.922914    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.922981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.922959    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.923082 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.923069    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.986568 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.986518    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmgzh\" (UniqueName: \"kubernetes.io/projected/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-kube-api-access-wmgzh\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:09.987223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:09.987203    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:10.101670 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:10.101638    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:10.529043 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:10.529003    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"]
Oct 30 22:02:10.530701 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:02:10.530668    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa7dcc5_d42a_4f74_8452_542b6ed5b404.slice/crio-337e8a54728561090b9d30e404f62b15b2bb8cdcbf0e2d7d16f30ffab05b1447 WatchSource:0}: Error finding container 337e8a54728561090b9d30e404f62b15b2bb8cdcbf0e2d7d16f30ffab05b1447: Status 404 returned error can't find the container with id 337e8a54728561090b9d30e404f62b15b2bb8cdcbf0e2d7d16f30ffab05b1447
Oct 30 22:02:10.533622 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:10.533593    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:02:10.533742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:10.533662    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:02:10.533742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:10.533697    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:02:10.705745 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:10.705705    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk" podUID="306879d4-4954-407f-b152-e1921381ac19" containerName="main" containerID="cri-o://90d2f9f62877017b2a62afdefd0464f3d58a8c80a0949569290df2665d10fd8c" gracePeriod=45
Oct 30 22:02:10.966915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:10.966874    4553 generic.go:356] "Generic (PLEG): container finished" podID="306879d4-4954-407f-b152-e1921381ac19" containerID="90d2f9f62877017b2a62afdefd0464f3d58a8c80a0949569290df2665d10fd8c" exitCode=0
Oct 30 22:02:10.967402 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:10.966940    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk" event={"ID":"306879d4-4954-407f-b152-e1921381ac19","Type":"ContainerDied","Data":"90d2f9f62877017b2a62afdefd0464f3d58a8c80a0949569290df2665d10fd8c"}
Oct 30 22:02:10.967402 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:10.966992    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk" event={"ID":"306879d4-4954-407f-b152-e1921381ac19","Type":"ContainerDied","Data":"c64bec7973c475efb505e498fc7d5410ca2972b6487cf2d39e623fa8a3219c98"}
Oct 30 22:02:10.967402 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:10.967009    4553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c64bec7973c475efb505e498fc7d5410ca2972b6487cf2d39e623fa8a3219c98"
Oct 30 22:02:10.969228 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:10.969202    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh" event={"ID":"9aa7dcc5-d42a-4f74-8452-542b6ed5b404","Type":"ContainerStarted","Data":"cac70d97ff7a51af1415f4f99c60ac2b942416a6ab116aae2ab65b45a24425f9"}
Oct 30 22:02:10.969386 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:10.969249    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh" event={"ID":"9aa7dcc5-d42a-4f74-8452-542b6ed5b404","Type":"ContainerStarted","Data":"337e8a54728561090b9d30e404f62b15b2bb8cdcbf0e2d7d16f30ffab05b1447"}
Oct 30 22:02:10.989434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:10.989409    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:02:11.038385 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.038348    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-tmp-dir\") pod \"306879d4-4954-407f-b152-e1921381ac19\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") "
Oct 30 22:02:11.038576 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.038493    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-home\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-home\") pod \"306879d4-4954-407f-b152-e1921381ac19\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") "
Oct 30 22:02:11.038576 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.038527    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/306879d4-4954-407f-b152-e1921381ac19-tls-certs\" (UniqueName: \"kubernetes.io/secret/306879d4-4954-407f-b152-e1921381ac19-tls-certs\") pod \"306879d4-4954-407f-b152-e1921381ac19\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") "
Oct 30 22:02:11.038696 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.038577    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-dshm\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-dshm\") pod \"306879d4-4954-407f-b152-e1921381ac19\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") "
Oct 30 22:02:11.038696 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.038649    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306879d4-4954-407f-b152-e1921381ac19-kube-api-access-7pbp5\" (UniqueName: \"kubernetes.io/projected/306879d4-4954-407f-b152-e1921381ac19-kube-api-access-7pbp5\") pod \"306879d4-4954-407f-b152-e1921381ac19\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") "
Oct 30 22:02:11.038696 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.038672    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-tmp-dir" pod "306879d4-4954-407f-b152-e1921381ac19" (UID: "306879d4-4954-407f-b152-e1921381ac19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:11.038847 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.038748    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-model-cache\") pod \"306879d4-4954-407f-b152-e1921381ac19\" (UID: \"306879d4-4954-407f-b152-e1921381ac19\") "
Oct 30 22:02:11.038847 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.038804    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-home" pod "306879d4-4954-407f-b152-e1921381ac19" (UID: "306879d4-4954-407f-b152-e1921381ac19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:11.039047 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.039021    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-model-cache" pod "306879d4-4954-407f-b152-e1921381ac19" (UID: "306879d4-4954-407f-b152-e1921381ac19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:11.039533 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.039509    4553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-model-cache\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:11.039641 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.039538    4553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-tmp-dir\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:11.039641 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.039554    4553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-home\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:11.041057 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.041033    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-dshm" pod "306879d4-4954-407f-b152-e1921381ac19" (UID: "306879d4-4954-407f-b152-e1921381ac19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:11.041225 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.041210    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/306879d4-4954-407f-b152-e1921381ac19-tls-certs" pod "306879d4-4954-407f-b152-e1921381ac19" (UID: "306879d4-4954-407f-b152-e1921381ac19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:11.041339 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.041318    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306879d4-4954-407f-b152-e1921381ac19-kube-api-access-7pbp5" pod "306879d4-4954-407f-b152-e1921381ac19" (UID: "306879d4-4954-407f-b152-e1921381ac19"). InnerVolumeSpecName "kube-api-access-7pbp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:11.102530 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.102499    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:11.103980 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.103956    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:02:11.104087 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.104004    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh" podUID="9aa7dcc5-d42a-4f74-8452-542b6ed5b404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 22:02:11.141051 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.140991    4553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306879d4-4954-407f-b152-e1921381ac19-tls-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:11.141372 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.141070    4553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306879d4-4954-407f-b152-e1921381ac19-dshm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:11.141372 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.141090    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pbp5\" (UniqueName: \"kubernetes.io/projected/306879d4-4954-407f-b152-e1921381ac19-kube-api-access-7pbp5\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:11.210170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.203289    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh" podStartSLOduration=2.203266534 podStartE2EDuration="2.203266534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:11.14167151 +0000 UTC m=+1181.876034540" watchObservedRunningTime="2026-10-30 22:02:11.203266534 +0000 UTC m=+1181.937629565"
Oct 30 22:02:11.974844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:11.974814    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"
Oct 30 22:02:12.102038 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:12.102005    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:02:12.102211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:12.102058    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh" podUID="9aa7dcc5-d42a-4f74-8452-542b6ed5b404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 22:02:12.142008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:12.141967    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"]
Oct 30 22:02:12.192275 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:12.192244    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/e2e-conflict-b-9b1cb2d8-kserve-97d5f54b8-qpdrk"]
Oct 30 22:02:13.102427 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:13.102394    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:02:13.102797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:13.102457    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh" podUID="9aa7dcc5-d42a-4f74-8452-542b6ed5b404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 22:02:13.393057 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:13.392978    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="306879d4-4954-407f-b152-e1921381ac19" path="/var/lib/kubelet/pods/306879d4-4954-407f-b152-e1921381ac19/volumes"
Oct 30 22:02:14.102868 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:14.102835    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:02:14.103359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:14.102909    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh" podUID="9aa7dcc5-d42a-4f74-8452-542b6ed5b404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 22:02:15.102661 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:15.102626    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:02:15.102876 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:15.102694    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh" podUID="9aa7dcc5-d42a-4f74-8452-542b6ed5b404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 22:02:16.102074 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:16.102035    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:02:16.102602 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:16.102092    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh" podUID="9aa7dcc5-d42a-4f74-8452-542b6ed5b404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 22:02:17.102353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:17.102308    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:02:17.102790 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:17.102392    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh" podUID="9aa7dcc5-d42a-4f74-8452-542b6ed5b404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 22:02:18.108229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:18.108195    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:18.108695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:18.108647    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:18.109420 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:18.109396    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:02:20.951339 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:20.951303    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-krsdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:02:20.951958 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:20.951921    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:20.993201 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:20.993155    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tdqzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:02:20.993550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:20.993517    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:21.401897 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.401857    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"]
Oct 30 22:02:21.402489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.402470    4553 cpu_manager.go:414] "containerMap: removing container" podUID="306879d4-4954-407f-b152-e1921381ac19" containerName="main"
Oct 30 22:02:21.402489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.402489    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306879d4-4954-407f-b152-e1921381ac19" containerName="main"
Oct 30 22:02:21.402660 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.402613    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="306879d4-4954-407f-b152-e1921381ac19" containerName="main"
Oct 30 22:02:21.405444 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.405423    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.458580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.458555    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1c3eef-openshift-default-dockercfg-hgpgr\""
Oct 30 22:02:21.513614 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.513576    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"]
Oct 30 22:02:21.546709 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.546572    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.546886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.546802    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.546948 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.546929    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.546997 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.546970    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.547129 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.547102    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.547225 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.547201    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.547280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.547232    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3267e456-b8e8-43be-9360-f75afeaadb7b-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.547339 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.547303    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.547430 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.547411    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcmgk\" (UniqueName: \"kubernetes.io/projected/3267e456-b8e8-43be-9360-f75afeaadb7b-kube-api-access-dcmgk\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.648667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.648631    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.648667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.648673    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.648922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.648723    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.648922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.648748    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.648922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.648767    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3267e456-b8e8-43be-9360-f75afeaadb7b-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.648922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.648797    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.648922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.648857    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcmgk\" (UniqueName: \"kubernetes.io/projected/3267e456-b8e8-43be-9360-f75afeaadb7b-kube-api-access-dcmgk\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.648922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.648920    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.649256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.648968    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.649256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.649157    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.649256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.649214    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.649734 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.649711    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.649849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.649753    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3267e456-b8e8-43be-9360-f75afeaadb7b-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.649911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.649876    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.651321 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.651302    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.651566 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.651548    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.708092 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.708011    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.708254 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.708107    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcmgk\" (UniqueName: \"kubernetes.io/projected/3267e456-b8e8-43be-9360-f75afeaadb7b-kube-api-access-dcmgk\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:21.719147 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:21.719111    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:22.161618 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:22.161590    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"]
Oct 30 22:02:22.162483 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:02:22.162458    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3267e456_b8e8_43be_9360_f75afeaadb7b.slice/crio-80c1dcbfb0cd2cec58022693ba0282444e140e23c0c37b68c36b86b5f9c1094a WatchSource:0}: Error finding container 80c1dcbfb0cd2cec58022693ba0282444e140e23c0c37b68c36b86b5f9c1094a: Status 404 returned error can't find the container with id 80c1dcbfb0cd2cec58022693ba0282444e140e23c0c37b68c36b86b5f9c1094a
Oct 30 22:02:22.164891 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:22.164860    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:02:22.164999 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:22.164924    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:02:22.164999 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:22.164955    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:02:23.107348 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:23.107311    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr" event={"ID":"3267e456-b8e8-43be-9360-f75afeaadb7b","Type":"ContainerStarted","Data":"a51d900eb15052a2ff2ebd314670c8d17ef846d2ddc232c189a5b7d835935f1c"}
Oct 30 22:02:23.107584 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:23.107376    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr" event={"ID":"3267e456-b8e8-43be-9360-f75afeaadb7b","Type":"ContainerStarted","Data":"80c1dcbfb0cd2cec58022693ba0282444e140e23c0c37b68c36b86b5f9c1094a"}
Oct 30 22:02:23.340924 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:23.340869    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr" podStartSLOduration=2.340854276 podStartE2EDuration="2.340854276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:23.288642195 +0000 UTC m=+1194.023005227" watchObservedRunningTime="2026-10-30 22:02:23.340854276 +0000 UTC m=+1194.075217307"
Oct 30 22:02:23.733159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:23.732620    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:23.740276 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:23.740243    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:02:23.740458 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:23.740303    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr" podUID="3267e456-b8e8-43be-9360-f75afeaadb7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 22:02:24.731561 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:24.731527    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:02:24.732188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:24.732151    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr" podUID="3267e456-b8e8-43be-9360-f75afeaadb7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 22:02:25.724603 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:25.724566    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:02:25.724915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:25.724883    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr" podUID="3267e456-b8e8-43be-9360-f75afeaadb7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 22:02:26.721631 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:26.721595    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:02:26.722284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:26.722251    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr" podUID="3267e456-b8e8-43be-9360-f75afeaadb7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 22:02:27.730159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:27.724511    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:02:27.730159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:27.724589    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr" podUID="3267e456-b8e8-43be-9360-f75afeaadb7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 22:02:28.201175 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:02:28.201101    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:02:28.720285 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:28.720249    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:02:28.720481 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:28.720312    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr" podUID="3267e456-b8e8-43be-9360-f75afeaadb7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 22:02:29.553531 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:29.553502    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:02:29.556211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:29.556189    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:02:29.724482 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:29.724453    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:29.724753 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:29.724736    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:29.725560 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:29.725540    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:02:35.958369 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:35.958293    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-krsdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:02:35.959076 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:35.959032    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:35.990169 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:35.986338    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tdqzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:02:35.990169 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:02:35.986410    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:58.202168 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:02:58.202114    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:03:14.531710 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.531679    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"]
Oct 30 22:03:14.535710 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.535693    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.577418 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.577388    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-dshm\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.577418 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.577427    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-tmp-dir\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.577617 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.577507    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-628kx\" (UniqueName: \"kubernetes.io/projected/1f2de714-796f-4938-8325-0004f2c22a02-kube-api-access-628kx\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.577659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.577636    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-model-cache\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.577693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.577678    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-home\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.577754 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.577738    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f2de714-796f-4938-8325-0004f2c22a02-tls-certs\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.639918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.639889    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b8717e\"/\"distinct-w0-b8717e-kserve-self-signed-certs\""
Oct 30 22:03:14.642615 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.642587    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"]
Oct 30 22:03:14.678607 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.678572    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-dshm\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.678865 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.678624    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-tmp-dir\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.678865 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.678686    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-628kx\" (UniqueName: \"kubernetes.io/projected/1f2de714-796f-4938-8325-0004f2c22a02-kube-api-access-628kx\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.678865 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.678793    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-model-cache\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.678865 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.678843    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-home\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.678865 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.678872    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f2de714-796f-4938-8325-0004f2c22a02-tls-certs\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.680459 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.680432    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-tmp-dir\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.680919 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.680895    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-model-cache\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.681380 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.681324    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-home\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.706171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.704950    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f2de714-796f-4938-8325-0004f2c22a02-tls-certs\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.733158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.729811    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-dshm\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.753002 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.752192    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-628kx\" (UniqueName: \"kubernetes.io/projected/1f2de714-796f-4938-8325-0004f2c22a02-kube-api-access-628kx\") pod \"distinct-w0-b8717e-kserve-544fffc6bc-kj7wc\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") " pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:14.848405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:14.848371    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:15.270333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:15.270297    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"]
Oct 30 22:03:15.270979 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:03:15.270953    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f2de714_796f_4938_8325_0004f2c22a02.slice/crio-46414c076b9beb17adecdf54f3135e4d71777ea218a871cda64b716c81dc8b5c WatchSource:0}: Error finding container 46414c076b9beb17adecdf54f3135e4d71777ea218a871cda64b716c81dc8b5c: Status 404 returned error can't find the container with id 46414c076b9beb17adecdf54f3135e4d71777ea218a871cda64b716c81dc8b5c
Oct 30 22:03:15.528596 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:15.528500    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" event={"ID":"1f2de714-796f-4938-8325-0004f2c22a02","Type":"ContainerStarted","Data":"7a8aeba4946931c896039d8dbc2c76f14d4462f350e74771cd2371f94460016a"}
Oct 30 22:03:15.528596 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:15.528545    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" event={"ID":"1f2de714-796f-4938-8325-0004f2c22a02","Type":"ContainerStarted","Data":"46414c076b9beb17adecdf54f3135e4d71777ea218a871cda64b716c81dc8b5c"}
Oct 30 22:03:15.647638 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:15.647580    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podStartSLOduration=1.647566947 podStartE2EDuration="1.647566947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:15.645106452 +0000 UTC m=+1246.379469483" watchObservedRunningTime="2026-10-30 22:03:15.647566947 +0000 UTC m=+1246.381929978"
Oct 30 22:03:15.849448 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:15.849353    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:16.778742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.778711    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"]
Oct 30 22:03:16.782671 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.782649    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.810958 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.810914    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.811171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.811127    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44783759-0591-4a2c-b850-97dff6f9085b-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.811296 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.811270    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.811347 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.811339    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.811397 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.811383    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgz6w\" (UniqueName: \"kubernetes.io/projected/44783759-0591-4a2c-b850-97dff6f9085b-kube-api-access-jgz6w\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.811547 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.811525    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44783759-0591-4a2c-b850-97dff6f9085b-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.811595 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.811559    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44783759-0591-4a2c-b850-97dff6f9085b-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.811653 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.811640    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.811697 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.811660    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.888856 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.888814    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"]
Oct 30 22:03:16.912708 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.912666    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.912708 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.912715    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.912947 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.912777    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.912947 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.912848    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44783759-0591-4a2c-b850-97dff6f9085b-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.912947 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.912874    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.912947 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.912906    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.913116 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.912975    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgz6w\" (UniqueName: \"kubernetes.io/projected/44783759-0591-4a2c-b850-97dff6f9085b-kube-api-access-jgz6w\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.913116 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.913085    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44783759-0591-4a2c-b850-97dff6f9085b-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.913238 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.913116    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44783759-0591-4a2c-b850-97dff6f9085b-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.913238 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.913156    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.913366 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.913350    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.913549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.913521    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.913691 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.913627    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.913781 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.913754    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44783759-0591-4a2c-b850-97dff6f9085b-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.915744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.915719    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.915975 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.915953    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44783759-0591-4a2c-b850-97dff6f9085b-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.971915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.971873    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44783759-0591-4a2c-b850-97dff6f9085b-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:16.972072 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:16.971933    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgz6w\" (UniqueName: \"kubernetes.io/projected/44783759-0591-4a2c-b850-97dff6f9085b-kube-api-access-jgz6w\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:17.097801 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:17.097775    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:17.525105 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:17.525076    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"]
Oct 30 22:03:17.525984 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:03:17.525952    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44783759_0591_4a2c_b850_97dff6f9085b.slice/crio-67f1304bc7d653404e43e227370b9a14395eab48a2f0e3d414ee19e6b16ccf3f WatchSource:0}: Error finding container 67f1304bc7d653404e43e227370b9a14395eab48a2f0e3d414ee19e6b16ccf3f: Status 404 returned error can't find the container with id 67f1304bc7d653404e43e227370b9a14395eab48a2f0e3d414ee19e6b16ccf3f
Oct 30 22:03:17.528395 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:17.528367    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:03:17.528471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:17.528432    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:03:17.528471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:17.528462    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:03:17.542247 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:17.542185    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7" event={"ID":"44783759-0591-4a2c-b850-97dff6f9085b","Type":"ContainerStarted","Data":"67f1304bc7d653404e43e227370b9a14395eab48a2f0e3d414ee19e6b16ccf3f"}
Oct 30 22:03:18.549921 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:18.549888    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7" event={"ID":"44783759-0591-4a2c-b850-97dff6f9085b","Type":"ContainerStarted","Data":"5481510b4d60c0f17ea910fd479c00adabb5430793c19cd89f62a9837cf32082"}
Oct 30 22:03:18.771766 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:18.771712    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7" podStartSLOduration=2.771697626 podStartE2EDuration="2.771697626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:18.720665244 +0000 UTC m=+1249.455028312" watchObservedRunningTime="2026-10-30 22:03:18.771697626 +0000 UTC m=+1249.506060657"
Oct 30 22:03:19.098942 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:19.098897    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:19.100427 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:19.100401    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:03:19.100577 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:19.100452    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7" podUID="44783759-0591-4a2c-b850-97dff6f9085b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 22:03:20.099121 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:20.099082    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:03:20.099558 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:20.099180    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7" podUID="44783759-0591-4a2c-b850-97dff6f9085b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 22:03:21.100718 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:21.100646    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:03:21.100718 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:21.100713    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7" podUID="44783759-0591-4a2c-b850-97dff6f9085b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 22:03:21.888214 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:21.888176    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"]
Oct 30 22:03:21.892387 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:21.892365    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:21.983486 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:21.983455    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:21.983665 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:21.983514    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:21.983665 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:21.983632    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:21.983740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:21.983692    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:21.983740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:21.983714    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:21.983800 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:21.983774    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:21.983840 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:21.983821    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gxtj\" (UniqueName: \"kubernetes.io/projected/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-kube-api-access-5gxtj\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:21.983879 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:21.983868    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:21.983942 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:21.983929    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:21.996534 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:21.996500    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"]
Oct 30 22:03:22.084903 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.084864    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.084903 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.084909    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.085233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.085033    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.085233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.085087    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gxtj\" (UniqueName: \"kubernetes.io/projected/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-kube-api-access-5gxtj\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.085233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.085112    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.085233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.085213    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.085419 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.085291    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.085419 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.085394    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.085508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.085458    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.085508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.085501    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.085644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.085613    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.085774 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.085754    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.085852 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.085833    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.085852 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.085840    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.087607 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.087587    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.087694 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.087645    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.098568 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.098547    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:03:22.098683 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.098615    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7" podUID="44783759-0591-4a2c-b850-97dff6f9085b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 22:03:22.148442 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.148369    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.148840 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.148505    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gxtj\" (UniqueName: \"kubernetes.io/projected/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-kube-api-access-5gxtj\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.206674 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.206636    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:22.660203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.660168    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"]
Oct 30 22:03:22.698289 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:03:22.698256    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c4dec5_f093_42e7_a8b1_68b5f2603a0c.slice/crio-5ff7f653c049788c128ce17e0de6b87b03236425d525327e45909420a540d9d5 WatchSource:0}: Error finding container 5ff7f653c049788c128ce17e0de6b87b03236425d525327e45909420a540d9d5: Status 404 returned error can't find the container with id 5ff7f653c049788c128ce17e0de6b87b03236425d525327e45909420a540d9d5
Oct 30 22:03:22.700663 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.700592    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:03:22.700742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.700664    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:03:22.700742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:22.700698    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:03:23.098568 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:23.098534    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:03:23.098752 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:23.098598    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7" podUID="44783759-0591-4a2c-b850-97dff6f9085b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 22:03:23.630824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:23.627710    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7" event={"ID":"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c","Type":"ContainerStarted","Data":"d7deea260d6770278fd6bbede79a18a8d2152eb18a0781bfe33b6ea5803ad8d1"}
Oct 30 22:03:23.630824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:23.627776    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7" event={"ID":"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c","Type":"ContainerStarted","Data":"5ff7f653c049788c128ce17e0de6b87b03236425d525327e45909420a540d9d5"}
Oct 30 22:03:23.760286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:23.760216    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7" podStartSLOduration=2.760192953 podStartE2EDuration="2.760192953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:23.757370447 +0000 UTC m=+1254.491733480" watchObservedRunningTime="2026-10-30 22:03:23.760192953 +0000 UTC m=+1254.494555986"
Oct 30 22:03:24.101166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:24.099539    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:03:24.101166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:24.099614    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7" podUID="44783759-0591-4a2c-b850-97dff6f9085b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 22:03:24.211448 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:24.211407    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:24.211626 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:24.211581    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:03:24.211708 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:24.211627    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7" podUID="b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 22:03:24.860161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:24.853429    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:24.898161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:24.897471    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:25.109161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:25.104598    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 22:03:25.109161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:25.104662    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7" podUID="44783759-0591-4a2c-b850-97dff6f9085b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 22:03:25.216221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:25.208522    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:03:25.216221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:25.208578    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7" podUID="b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 22:03:26.099665 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:26.095853    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:26.208096 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:26.208026    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:03:26.208304 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:26.208118    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7" podUID="b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 22:03:26.265161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:26.230458    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:26.265161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:26.231931    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:26.269188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:26.266492    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:03:27.207599 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:27.207562    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:03:27.208204 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:27.208168    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7" podUID="b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 22:03:28.202114 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:03:28.202073    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:03:28.207535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:28.207508    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:03:28.207681 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:28.207596    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7" podUID="b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 22:03:29.207394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:29.207355    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 30 22:03:29.207670 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:29.207420    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7" podUID="b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 30 22:03:30.663936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:30.663899    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:30.664486 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:30.664466    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:30.665417 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:30.665398    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:03:47.711295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:47.711243    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:47.817290 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:47.817237    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:47.979586 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:47.979483    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 22:03:48.847161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:48.846291    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:03:48.847161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:48.846715    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:48.881971 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:48.880221    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:03:48.881971 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:48.880667    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 22:03:48.898690 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:48.898264    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:48.933650 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:48.933608    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:49.058163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:49.030939    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 22:03:49.118217 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:49.099477    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 22:03:49.221284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:49.221242    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:03:49.296171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:49.296090    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh" podUID="9aa7dcc5-d42a-4f74-8452-542b6ed5b404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:49.849420 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:49.849383    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:49.931295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:49.931243    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:49.932609 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:49.932578    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:50.080390 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:50.022431    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:50.098531 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:50.098479    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hfxrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:03:50.102534 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:50.102437    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj" podUID="20775493-c504-4316-9d6f-816952c83e9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:50.106313 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:50.102353    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 22:03:50.221162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:50.218994    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:03:50.221162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:50.219070    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:03:50.971778 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:50.971732    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:03:51.030200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:51.020554    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-krsdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:03:51.030200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:51.020618    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:51.071059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:51.043342    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:03:51.147181 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:51.120524    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tdqzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:03:51.147181 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:51.120612    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:51.147181 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:51.120745    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:51.147181 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:51.121273    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:03:51.147181 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:51.123456    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 30 22:03:51.665612 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:51.665579    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:03:51.671926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:51.671902    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:03:51.673072 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:51.673051    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 22:03:51.708727 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:51.708697    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 22:03:51.876797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:51.876767    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:03:51.948326 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:51.948232    4553 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:52.409440 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.409399    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"]
Oct 30 22:03:52.413744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.413722    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.437493 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.437464    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"]
Oct 30 22:03:52.441556 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.441535    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.481489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.481453    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"]
Oct 30 22:03:52.487037 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.486995    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6zj6\" (UniqueName: \"kubernetes.io/projected/fa482183-5d9f-461a-af5d-b30fd2ed8428-kube-api-access-b6zj6\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.487232 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487078    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.487232 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487126    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.487322 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487235    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.487322 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487277    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.487322 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487306    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.487470 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487351    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.487470 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487416    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.487470 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487464    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlzwd\" (UniqueName: \"kubernetes.io/projected/4bb761f4-75aa-481c-ad0a-0cfc782028ac-kube-api-access-tlzwd\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.487636 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487506    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.487636 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487529    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.487636 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487567    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.487636 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487602    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.487636 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487625    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.488411 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487662    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.488411 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487713    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.488411 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487764    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.488411 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.487830    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa482183-5d9f-461a-af5d-b30fd2ed8428-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.550000 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.549951    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"]
Oct 30 22:03:52.589060 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589029    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589070    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589089    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589112    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589160    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589193    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589246    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa482183-5d9f-461a-af5d-b30fd2ed8428-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589316    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6zj6\" (UniqueName: \"kubernetes.io/projected/fa482183-5d9f-461a-af5d-b30fd2ed8428-kube-api-access-b6zj6\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589364    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589419    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589444    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589491    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589522    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589569    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589634    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589687    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlzwd\" (UniqueName: \"kubernetes.io/projected/4bb761f4-75aa-481c-ad0a-0cfc782028ac-kube-api-access-tlzwd\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589729    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589755    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.590110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.589753    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.591211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.590121    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.591211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.590303    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.591211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.590639    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.591211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.590686    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.591211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.590801    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.591211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.590839    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.591211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.590918    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.591211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.591080    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.591211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.591101    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa482183-5d9f-461a-af5d-b30fd2ed8428-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.592597 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.592220    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.593378 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.593354    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.593725 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.593703    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.594419 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.594400    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.648492 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.648459    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.649091 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.649061    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlzwd\" (UniqueName: \"kubernetes.io/projected/4bb761f4-75aa-481c-ad0a-0cfc782028ac-kube-api-access-tlzwd\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.649254 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.649167    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.650238 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.650220    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6zj6\" (UniqueName: \"kubernetes.io/projected/fa482183-5d9f-461a-af5d-b30fd2ed8428-kube-api-access-b6zj6\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:52.738278 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.737269    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:52.763252 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:52.762967    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:53.192969 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.192942    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"]
Oct 30 22:03:53.194220 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:03:53.194194    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb761f4_75aa_481c_ad0a_0cfc782028ac.slice/crio-8f6a8b7a9037bfef2101bd45d9bd2a9ca8d80843b973688655c0be6f4f82175e WatchSource:0}: Error finding container 8f6a8b7a9037bfef2101bd45d9bd2a9ca8d80843b973688655c0be6f4f82175e: Status 404 returned error can't find the container with id 8f6a8b7a9037bfef2101bd45d9bd2a9ca8d80843b973688655c0be6f4f82175e
Oct 30 22:03:53.194804 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:03:53.194778    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa482183_5d9f_461a_af5d_b30fd2ed8428.slice/crio-bef109eff1fea61484f26d34fed8026102b02d43d139095337498003412cb5e6 WatchSource:0}: Error finding container bef109eff1fea61484f26d34fed8026102b02d43d139095337498003412cb5e6: Status 404 returned error can't find the container with id bef109eff1fea61484f26d34fed8026102b02d43d139095337498003412cb5e6
Oct 30 22:03:53.196806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.196773    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:03:53.196917 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.196845    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:03:53.196917 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.196880    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:03:53.197190 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.197125    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:03:53.197245 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.197226    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:03:53.197285 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.197254    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:03:53.242531 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.242504    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"]
Oct 30 22:03:53.689447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.689411    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp" event={"ID":"fa482183-5d9f-461a-af5d-b30fd2ed8428","Type":"ContainerStarted","Data":"e9ba3d7bc6a24adbc52da5c485bd54a4d072889f45276618a26e740b76244f85"}
Oct 30 22:03:53.689948 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.689483    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp" event={"ID":"fa482183-5d9f-461a-af5d-b30fd2ed8428","Type":"ContainerStarted","Data":"bef109eff1fea61484f26d34fed8026102b02d43d139095337498003412cb5e6"}
Oct 30 22:03:53.691229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.691203    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n" event={"ID":"4bb761f4-75aa-481c-ad0a-0cfc782028ac","Type":"ContainerStarted","Data":"509b5b63acc19168ffc8d8fd98093ce9f9c2462a18c2e495b4b143667c7ee5b7"}
Oct 30 22:03:53.691345 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.691252    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n" event={"ID":"4bb761f4-75aa-481c-ad0a-0cfc782028ac","Type":"ContainerStarted","Data":"8f6a8b7a9037bfef2101bd45d9bd2a9ca8d80843b973688655c0be6f4f82175e"}
Oct 30 22:03:53.745158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.740220    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:03:53.767579 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.764455    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:03:53.817098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.813942    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp" podStartSLOduration=1.813920069 podStartE2EDuration="1.813920069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:53.811872843 +0000 UTC m=+1284.546235942" watchObservedRunningTime="2026-10-30 22:03:53.813920069 +0000 UTC m=+1284.548283101"
Oct 30 22:03:53.958221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:53.956363    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n" podStartSLOduration=1.9563381469999999 podStartE2EDuration="1.956338147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:53.936475647 +0000 UTC m=+1284.670838679" watchObservedRunningTime="2026-10-30 22:03:53.956338147 +0000 UTC m=+1284.690701171"
Oct 30 22:03:54.759082 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:54.758750    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:03:54.759082 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:54.758809    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n" podUID="4bb761f4-75aa-481c-ad0a-0cfc782028ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:03:54.765439 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:54.765403    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:03:54.765587 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:54.765460    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp" podUID="fa482183-5d9f-461a-af5d-b30fd2ed8428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:03:55.747163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:55.741517    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:03:55.747163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:55.741577    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n" podUID="4bb761f4-75aa-481c-ad0a-0cfc782028ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:03:55.764391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:55.764363    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:03:55.764879 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:55.764853    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp" podUID="fa482183-5d9f-461a-af5d-b30fd2ed8428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:03:56.740159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:56.739513    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:03:56.740159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:56.739577    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n" podUID="4bb761f4-75aa-481c-ad0a-0cfc782028ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:03:56.765076 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:56.765030    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:03:56.765710 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:56.765682    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp" podUID="fa482183-5d9f-461a-af5d-b30fd2ed8428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:03:57.738449 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:57.738414    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:03:57.739242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:57.739205    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n" podUID="4bb761f4-75aa-481c-ad0a-0cfc782028ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:03:57.764976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:57.764938    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:03:57.765260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:57.765011    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp" podUID="fa482183-5d9f-461a-af5d-b30fd2ed8428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:03:58.203349 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:03:58.203290    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:03:58.740157 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:58.739535    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:03:58.740157 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:58.739613    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n" podUID="4bb761f4-75aa-481c-ad0a-0cfc782028ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:03:58.764160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:58.764112    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:03:58.764355 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:58.764191    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp" podUID="fa482183-5d9f-461a-af5d-b30fd2ed8428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:03:59.738091 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:59.738049    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:03:59.738500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:59.738157    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n" podUID="4bb761f4-75aa-481c-ad0a-0cfc782028ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:03:59.764519 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:59.764486    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:03:59.764700 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:03:59.764543    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp" podUID="fa482183-5d9f-461a-af5d-b30fd2ed8428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:04:00.742866 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:00.742836    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:04:00.743450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:00.743315    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:04:00.743977 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:00.743956    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:04:00.776477 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:00.776446    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:04:00.776921 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:00.776884    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:04:00.779732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:00.779707    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:04:02.204146 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.204105    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"]
Oct 30 22:04:02.209650 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.209627    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.302393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.302358    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.302579 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.302417    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.302579 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.302539    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.302749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.302724    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.302806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.302789    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.302903 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.302887    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.302979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.302915    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.302979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.302939    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzgjd\" (UniqueName: \"kubernetes.io/projected/49818a62-7ddf-43b3-9aa8-cf0aadf73516-kube-api-access-rzgjd\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.303179 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.302985    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.318798 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.318764    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"]
Oct 30 22:04:02.404423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.404386    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.404622 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.404527    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.404622 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.404567    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.404715 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.404644    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.404715 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.404668    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.404715 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.404691    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzgjd\" (UniqueName: \"kubernetes.io/projected/49818a62-7ddf-43b3-9aa8-cf0aadf73516-kube-api-access-rzgjd\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.404873 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.404735    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.404873 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.404769    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.404873 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.404811    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.405061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.405035    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.405194 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.405168    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.405273 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.405189    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.405313 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.405267    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.405504 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.405473    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.407359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.407334    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.407572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.407555    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.463860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.463792    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.464013 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.463912    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzgjd\" (UniqueName: \"kubernetes.io/projected/49818a62-7ddf-43b3-9aa8-cf0aadf73516-kube-api-access-rzgjd\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.523526 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.523489    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:02.954591 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.954564    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"]
Oct 30 22:04:02.955531 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:04:02.955507    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49818a62_7ddf_43b3_9aa8_cf0aadf73516.slice/crio-89e595557c3e04f3ad9f7fddc4a71ca7f56c6782384d9bb842ec977cfc309862 WatchSource:0}: Error finding container 89e595557c3e04f3ad9f7fddc4a71ca7f56c6782384d9bb842ec977cfc309862: Status 404 returned error can't find the container with id 89e595557c3e04f3ad9f7fddc4a71ca7f56c6782384d9bb842ec977cfc309862
Oct 30 22:04:02.958672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.958638    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:04:02.958792 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.958721    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:04:02.958792 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:02.958770    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:04:03.805907 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:03.805875    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp" event={"ID":"49818a62-7ddf-43b3-9aa8-cf0aadf73516","Type":"ContainerStarted","Data":"7799182f07e44817b9391a7954c253083b73455deafa5d1bf674396bd1b86854"}
Oct 30 22:04:03.806375 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:03.805928    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp" event={"ID":"49818a62-7ddf-43b3-9aa8-cf0aadf73516","Type":"ContainerStarted","Data":"89e595557c3e04f3ad9f7fddc4a71ca7f56c6782384d9bb842ec977cfc309862"}
Oct 30 22:04:03.993794 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:03.993736    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp" podStartSLOduration=1.99371947 podStartE2EDuration="1.99371947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:03.991722394 +0000 UTC m=+1294.726085452" watchObservedRunningTime="2026-10-30 22:04:03.99371947 +0000 UTC m=+1294.728082570"
Oct 30 22:04:04.525974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:04.525940    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:04.533354 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:04.533314    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:04:04.533586 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:04.533378    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp" podUID="49818a62-7ddf-43b3-9aa8-cf0aadf73516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 22:04:05.526161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:05.525616    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:04:05.526161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:05.525686    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp" podUID="49818a62-7ddf-43b3-9aa8-cf0aadf73516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 22:04:06.530161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:06.529282    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:04:06.530161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:06.529363    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp" podUID="49818a62-7ddf-43b3-9aa8-cf0aadf73516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 22:04:07.260942 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.260903    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"]
Oct 30 22:04:07.265615 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.265590    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.373409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.373377    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"]
Oct 30 22:04:07.387659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.387618    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.387836 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.387679    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.387836 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.387730    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.387836 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.387811    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.387967 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.387860    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.387967 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.387956    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.388046 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.387982    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgk5x\" (UniqueName: \"kubernetes.io/projected/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-kube-api-access-pgk5x\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.388125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.388109    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.388221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.388201    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.489198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.489123    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.489400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.489211    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.489400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.489307    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.489400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.489331    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgk5x\" (UniqueName: \"kubernetes.io/projected/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-kube-api-access-pgk5x\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.489576 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.489449    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.489703 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.489683    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.489872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.489849    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.490017 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.489899    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.490017 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.489937    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.490017 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.489969    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-workload-certs\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.490251 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.490059    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istiod-ca-cert\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.490381 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.490357    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-credential-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.490442 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.490383    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-data\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.490621 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.490598    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-workload-socket\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.491702 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.491678    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-envoy\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.492250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.492224    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-podinfo\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.533218 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.532671    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:04:07.533218 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.532770    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp" podUID="49818a62-7ddf-43b3-9aa8-cf0aadf73516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 22:04:07.548184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.548154    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-token\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.548370 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.548347    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgk5x\" (UniqueName: \"kubernetes.io/projected/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-kube-api-access-pgk5x\") pod \"e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") " pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:07.581267 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:07.581221    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:08.046046 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:08.046012    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"]
Oct 30 22:04:08.046408 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:04:08.046380    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f1880e_2463_4df9_9c57_1d0dabe76c4b.slice/crio-73309d2b766f33805e67a480eb7693d22c73dbe251e9ab550f5e4743c69abcbd WatchSource:0}: Error finding container 73309d2b766f33805e67a480eb7693d22c73dbe251e9ab550f5e4743c69abcbd: Status 404 returned error can't find the container with id 73309d2b766f33805e67a480eb7693d22c73dbe251e9ab550f5e4743c69abcbd
Oct 30 22:04:08.049179 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:08.049126    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:04:08.049252 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:08.049227    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:04:08.049299 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:08.049273    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:04:08.525611 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:08.525571    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:04:08.525895 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:08.525865    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp" podUID="49818a62-7ddf-43b3-9aa8-cf0aadf73516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 22:04:08.871037 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:08.870947    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" event={"ID":"c2f1880e-2463-4df9-9c57-1d0dabe76c4b","Type":"ContainerStarted","Data":"09adb8417253ca932ad9f4ff3d73181299ed2519c9df3acd9a3f38393ed11493"}
Oct 30 22:04:08.871037 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:08.871036    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" event={"ID":"c2f1880e-2463-4df9-9c57-1d0dabe76c4b","Type":"ContainerStarted","Data":"73309d2b766f33805e67a480eb7693d22c73dbe251e9ab550f5e4743c69abcbd"}
Oct 30 22:04:08.996429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:08.996363    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podStartSLOduration=1.9963438610000002 podStartE2EDuration="1.996343861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:08.995317734 +0000 UTC m=+1299.729680766" watchObservedRunningTime="2026-10-30 22:04:08.996343861 +0000 UTC m=+1299.730706893"
Oct 30 22:04:09.525161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:09.524626    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 30 22:04:09.525161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:09.524694    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp" podUID="49818a62-7ddf-43b3-9aa8-cf0aadf73516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 30 22:04:09.581873 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:09.581836    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:09.583603 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:09.583577    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:04:09.583733 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:09.583633    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 22:04:10.540632 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.540592    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"]
Oct 30 22:04:10.545321 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.545297    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.582515 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.582479    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:04:10.582710 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.582542    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 22:04:10.644869 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.644822    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"]
Oct 30 22:04:10.672442 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.672405    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a57febb1-62d4-475d-8a12-61cad4e3b93a-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.672625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.672472    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.672625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.672514    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.672625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.672590    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.672625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.672618    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.672842 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.672640    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.672842 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.672656    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbxft\" (UniqueName: \"kubernetes.io/projected/a57febb1-62d4-475d-8a12-61cad4e3b93a-kube-api-access-lbxft\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.672842 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.672697    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.672842 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.672747    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.773698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.773660    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.773995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.773969    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.774188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.774127    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.774329 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.774309    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbxft\" (UniqueName: \"kubernetes.io/projected/a57febb1-62d4-475d-8a12-61cad4e3b93a-kube-api-access-lbxft\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.774495 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.774478    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.774638 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.774614    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-credential-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.774835 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.774649    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.774976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.774949    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-workload-socket\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.775545 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.775017    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a57febb1-62d4-475d-8a12-61cad4e3b93a-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.775545 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.775251    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.775545 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.775333    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.775778 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.775699    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a57febb1-62d4-475d-8a12-61cad4e3b93a-istiod-ca-cert\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.775778 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.775697    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-data\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.777000 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.776976    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-workload-certs\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.779871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.779788    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-envoy\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.780523 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.780478    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-podinfo\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.852040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.851961    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbxft\" (UniqueName: \"kubernetes.io/projected/a57febb1-62d4-475d-8a12-61cad4e3b93a-kube-api-access-lbxft\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.853199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.853110    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-token\") pod \"e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") " pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.865338 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.865014    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:10.963672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.960843    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:10.963855 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.963809    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:10.965111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:10.965090    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:04:11.350295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:11.350261    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"]
Oct 30 22:04:11.351248 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:04:11.351226    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57febb1_62d4_475d_8a12_61cad4e3b93a.slice/crio-0ceb10c9e511052b40d79995d27d0ca06bfb6ebb3f3152b4415d13b7535864ba WatchSource:0}: Error finding container 0ceb10c9e511052b40d79995d27d0ca06bfb6ebb3f3152b4415d13b7535864ba: Status 404 returned error can't find the container with id 0ceb10c9e511052b40d79995d27d0ca06bfb6ebb3f3152b4415d13b7535864ba
Oct 30 22:04:11.354032 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:11.353996    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:04:11.354167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:11.354072    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:04:11.354167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:11.354102    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:04:11.582948 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:11.582913    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:04:11.583636 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:11.583604    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 22:04:12.377170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:12.367885    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" event={"ID":"a57febb1-62d4-475d-8a12-61cad4e3b93a","Type":"ContainerStarted","Data":"2befb8b0a81b279f2fb31c1bc180faa0e30de8d22cd5d4258134cf6e3119d9dc"}
Oct 30 22:04:12.377561 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:12.377520    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" event={"ID":"a57febb1-62d4-475d-8a12-61cad4e3b93a","Type":"ContainerStarted","Data":"0ceb10c9e511052b40d79995d27d0ca06bfb6ebb3f3152b4415d13b7535864ba"}
Oct 30 22:04:12.589353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:12.589265    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:04:12.601989 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:12.601947    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 22:04:12.768327 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:12.767319    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:12.882163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:12.881826    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:12.882163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:12.882031    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:04:12.882163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:12.882081    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:04:12.914160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:12.903689    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.074753 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:13.074623    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.109160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:13.102944    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.231282 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:13.231231    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.584599 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:13.584560    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:04:13.584890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:13.584859    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 22:04:13.776160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:13.769217    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.776160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:13.769443    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 22:04:13.884226 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:13.870080    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:04:13.884226 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:13.870157    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:04:13.919155 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:13.909524    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" podStartSLOduration=3.909506053 podStartE2EDuration="3.909506053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:12.513765799 +0000 UTC m=+1303.248128833" watchObservedRunningTime="2026-10-30 22:04:13.909506053 +0000 UTC m=+1304.643869086"
Oct 30 22:04:13.955163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:13.954249    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:13.955163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:13.954650    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:04:14.091092 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:14.091030    4553 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:14.179773 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:14.108383    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:14.179773 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:14.108863    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:04:14.179773 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:14.137794    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.179773 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:14.140720    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 22:04:14.735161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:14.730658    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:04:14.735161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:14.730731    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 22:04:14.893448 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:14.892211    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:14.919221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:14.908184    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:04:14.919221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:14.908244    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:04:15.005263 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:14.966473    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.142162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:15.141308    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.168162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:15.125217    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.168162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:15.141258    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.585573 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:15.585536    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:04:15.589160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:15.585823    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 22:04:15.629161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:15.615502    4553 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.813171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:15.800291    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:15.813171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:15.800422    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr" podUID="3267e456-b8e8-43be-9360-f75afeaadb7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.875833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:15.875630    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:04:15.875833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:15.875705    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:04:15.900280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:15.900231    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 22:04:16.037163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:15.978703    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.147350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.147237    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.160165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.155596    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.186160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.185506    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:04:16.186160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.185915    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:04:16.196626 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.196489    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:16.196626 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.196552    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q" podUID="5c189afe-afe8-4736-8e80-e321da9d7268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.653540 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.653501    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:04:16.721167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.710752    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 22:04:16.843282 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.825557    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:16.849161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.843535    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7" podUID="b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.849473 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.846838    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:16.849635 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.849605    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp" podUID="fa482183-5d9f-461a-af5d-b30fd2ed8428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.858163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.850547    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:16.858163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.851371    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n" podUID="4bb761f4-75aa-481c-ad0a-0cfc782028ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.892687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.892645    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:04:16.900191 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.896254    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:04:16.913309 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.900911    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 22:04:16.999385 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:16.999335    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.183221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:17.177541    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.183221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:17.177957    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.203427 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:17.203377    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:04:17.522222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:17.509293    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:17.522804 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:17.522746    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b" podUID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.656345 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:17.656307    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:04:17.656840 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:17.656802    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 22:04:17.924277 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:17.924221    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 22:04:17.990307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:17.949288    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:04:17.991569 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:17.991504    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:04:18.017376 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:18.004377    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.290673 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:18.290570    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:04:18.291467 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:18.291227    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:04:18.299219 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:18.291304    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:18.391449 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:18.391417    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 22:04:18.402382 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:18.401504    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:04:18.414581 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:18.414548    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:04:18.419268 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:18.419243    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:04:18.464753 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:18.464447    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 22:04:18.582594 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:18.582569    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:04:18.582732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:18.582617    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 22:04:18.872374 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:18.872273    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:04:18.872374 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:18.872339    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:04:19.582279 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:19.582246    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 22:04:19.582677 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:19.582296    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 22:04:19.865686 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:19.865606    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:04:19.865686 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:19.865660    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:04:20.867521 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:20.867486    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:04:20.868005 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:20.867538    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:04:21.583935 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:21.583884    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:21.584195 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:21.583956    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:21.865838 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:21.865735    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:04:21.865838 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:21.865787    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:04:22.582345 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:22.582310    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:22.625380 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:22.625343    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:22.625563 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:22.625418    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:22.712595 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:22.712473    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:22.875158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:22.869827    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 30 22:04:22.875158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:22.869994    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 30 22:04:23.184360 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:23.184335    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:04:24.102710 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:24.102678    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:24.103317 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:24.103180    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:24.104261 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:24.104240    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:04:28.203930 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:04:28.203877    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:04:32.533188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.533151    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"]
Oct 30 22:04:32.538745 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.538720    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.552301 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.552272    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.552301 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.552303    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdkr8\" (UniqueName: \"kubernetes.io/projected/eedadc43-2c7f-465a-a834-f1871f020e61-kube-api-access-cdkr8\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.552527 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.552329    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.552527 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.552500    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.552631 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.552552    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedadc43-2c7f-465a-a834-f1871f020e61-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.552688 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.552661    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedadc43-2c7f-465a-a834-f1871f020e61-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.552821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.552804    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.552946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.552932    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedadc43-2c7f-465a-a834-f1871f020e61-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.552993 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.552983    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.575645 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.575612    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"]
Oct 30 22:04:32.653795 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.653751    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.654001 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.653882    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedadc43-2c7f-465a-a834-f1871f020e61-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.654001 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.653917    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.654001 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.653957    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.654001 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.653979    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdkr8\" (UniqueName: \"kubernetes.io/projected/eedadc43-2c7f-465a-a834-f1871f020e61-kube-api-access-cdkr8\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.654323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.654008    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.654323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.654067    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.654323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.654104    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedadc43-2c7f-465a-a834-f1871f020e61-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.654323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.654162    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedadc43-2c7f-465a-a834-f1871f020e61-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.654323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.654240    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-istio-data\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.654323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.654307    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-workload-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.654627 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.654457    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-credential-socket\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.654627 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.654513    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-workload-certs\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.654930 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.654906    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedadc43-2c7f-465a-a834-f1871f020e61-istiod-ca-cert\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.656564 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.656539    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-istio-envoy\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.656790 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.656772    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedadc43-2c7f-465a-a834-f1871f020e61-istio-podinfo\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.713047 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.713017    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedadc43-2c7f-465a-a834-f1871f020e61-istio-token\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.713367 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.713342    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdkr8\" (UniqueName: \"kubernetes.io/projected/eedadc43-2c7f-465a-a834-f1871f020e61-kube-api-access-cdkr8\") pod \"e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") " pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:32.855470 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:32.855433    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:33.288391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:33.288364    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"]
Oct 30 22:04:33.289191 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:04:33.289160    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedadc43_2c7f_465a_a834_f1871f020e61.slice/crio-9ecfe3ffc8733aa218425faed2708cff2acb61f6bad521adf94d165a5ade9d29 WatchSource:0}: Error finding container 9ecfe3ffc8733aa218425faed2708cff2acb61f6bad521adf94d165a5ade9d29: Status 404 returned error can't find the container with id 9ecfe3ffc8733aa218425faed2708cff2acb61f6bad521adf94d165a5ade9d29
Oct 30 22:04:33.291700 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:33.291671    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:04:33.291772 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:33.291733    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:04:33.291772 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:33.291759    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:04:34.261576 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:34.261544    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" event={"ID":"eedadc43-2c7f-465a-a834-f1871f020e61","Type":"ContainerStarted","Data":"a77e9f6fce43fbde3e46a85d21f6d36039152f2cd870607296c609d18d778f91"}
Oct 30 22:04:34.262063 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:34.261604    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" event={"ID":"eedadc43-2c7f-465a-a834-f1871f020e61","Type":"ContainerStarted","Data":"9ecfe3ffc8733aa218425faed2708cff2acb61f6bad521adf94d165a5ade9d29"}
Oct 30 22:04:34.390666 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:34.390607    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" podStartSLOduration=2.390589552 podStartE2EDuration="2.390589552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:34.388776708 +0000 UTC m=+1325.123139749" watchObservedRunningTime="2026-10-30 22:04:34.390589552 +0000 UTC m=+1325.124952583"
Oct 30 22:04:34.872928 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:34.872897    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:34.874314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:34.874281    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:34.874520 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:34.874486    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" podUID="eedadc43-2c7f-465a-a834-f1871f020e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:04:35.857811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:35.857768    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:35.859726 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:35.859690    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" podUID="eedadc43-2c7f-465a-a834-f1871f020e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:04:36.174225 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:36.169233    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:36.190762 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:36.190715    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:36.261163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:36.253830    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:36.867272 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:36.867233    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:36.867972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:36.867940    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" podUID="eedadc43-2c7f-465a-a834-f1871f020e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:04:36.967787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:36.967744    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:36.973272 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:36.973210    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:37.179360 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:37.179240    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:37.197164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:37.188667    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:04:37.229162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:37.213234    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:04:37.229162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:37.213695    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 22:04:37.272272 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:37.272223    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:37.280198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:37.272969    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:04:37.896162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:37.892297    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:37.896162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:37.892362    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" podUID="eedadc43-2c7f-465a-a834-f1871f020e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:04:37.973342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:37.973297    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:38.025309 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.025272    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 22:04:38.051161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.025740    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:38.110163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.069903    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"]
Oct 30 22:04:38.205227 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.192603    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Oct 30 22:04:38.213288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.198388    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:04:38.247165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.198722    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.273783 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.267216    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:04:38.274072 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.274037    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:38.453563 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.432623    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.547812 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.547710    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.548176 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.548115    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.548340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.548321    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp7xz\" (UniqueName: \"kubernetes.io/projected/3e4c8521-f6ec-4d79-8946-045048bdad39-kube-api-access-tp7xz\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.548500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.548483    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.548742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.548724    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4c8521-f6ec-4d79-8946-045048bdad39-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.548906 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.548888    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.550890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.549108    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.551206 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.551162    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.572257 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.572212    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"]
Oct 30 22:04:38.613669 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.613635    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:04:38.728168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.667636    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.868162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.853237    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp7xz\" (UniqueName: \"kubernetes.io/projected/3e4c8521-f6ec-4d79-8946-045048bdad39-kube-api-access-tp7xz\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.868162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.853348    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.868162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.853475    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4c8521-f6ec-4d79-8946-045048bdad39-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.868162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.853536    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.868162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.853647    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.868162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.853690    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.868162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.853754    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.868162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.853816    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.868162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.854268    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.868162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.723247    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.868162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.854881    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.868162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.855527    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4c8521-f6ec-4d79-8946-045048bdad39-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.868162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.855772    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:38.901161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.897452    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:04:38.929433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.929401    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:04:38.968163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.907310    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:39.041288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:39.041235    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" podUID="eedadc43-2c7f-465a-a834-f1871f020e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:04:39.041561 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:39.004306    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:39.046214 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:38.990836    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:39.068170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:39.007034    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp7xz\" (UniqueName: \"kubernetes.io/projected/3e4c8521-f6ec-4d79-8946-045048bdad39-kube-api-access-tp7xz\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:39.195566 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:39.195458    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:39.195873 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:39.195837    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:39.259904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:39.233824    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:39.279899 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:39.279846    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:39.376557 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:39.376399    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:39.503633 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:39.503119    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:39.564258 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:39.563716    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 22:04:39.864582 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:39.864548    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:39.864853 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:39.864818    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" podUID="eedadc43-2c7f-465a-a834-f1871f020e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:04:39.880937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:39.879850    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 22:04:40.302541 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:40.302511    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"]
Oct 30 22:04:40.304076 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:04:40.304053    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4c8521_f6ec_4d79_8946_045048bdad39.slice/crio-d5c610a0b3929cff23bfa1ee2fbd3022af156f2d2cb182d5d100b42ebaadfb42 WatchSource:0}: Error finding container d5c610a0b3929cff23bfa1ee2fbd3022af156f2d2cb182d5d100b42ebaadfb42: Status 404 returned error can't find the container with id d5c610a0b3929cff23bfa1ee2fbd3022af156f2d2cb182d5d100b42ebaadfb42
Oct 30 22:04:40.306589 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:40.306557    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:04:40.306712 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:40.306639    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:04:40.306712 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:40.306688    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:04:40.877783 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:40.876439    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:40.877783 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:40.876496    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" podUID="eedadc43-2c7f-465a-a834-f1871f020e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:04:40.984325 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:40.974732    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr" event={"ID":"3e4c8521-f6ec-4d79-8946-045048bdad39","Type":"ContainerStarted","Data":"39f6245dd87a5e20de41fd6e8f0012d20d66d2b8f8159277760d2c423c4923d1"}
Oct 30 22:04:40.984325 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:40.974806    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr" event={"ID":"3e4c8521-f6ec-4d79-8946-045048bdad39","Type":"ContainerStarted","Data":"d5c610a0b3929cff23bfa1ee2fbd3022af156f2d2cb182d5d100b42ebaadfb42"}
Oct 30 22:04:41.197242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:41.194240    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr" podStartSLOduration=4.194213734 podStartE2EDuration="4.194213734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:41.168503588 +0000 UTC m=+1331.902866621" watchObservedRunningTime="2026-10-30 22:04:41.194213734 +0000 UTC m=+1331.928576766"
Oct 30 22:04:41.506904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:41.506372    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:41.507745 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:41.507716    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:41.507881 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:41.507777    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr" podUID="3e4c8521-f6ec-4d79-8946-045048bdad39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:04:41.856157 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:41.856019    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:41.856157 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:41.856077    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" podUID="eedadc43-2c7f-465a-a834-f1871f020e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:04:42.506604 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:42.506568    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:42.506783 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:42.506638    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr" podUID="3e4c8521-f6ec-4d79-8946-045048bdad39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:04:42.866160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:42.858730    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:04:42.866160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:42.858792    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" podUID="eedadc43-2c7f-465a-a834-f1871f020e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:04:43.506302 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:43.506257    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:43.506505 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:43.506328    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr" podUID="3e4c8521-f6ec-4d79-8946-045048bdad39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:04:44.506907 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:44.506874    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:44.507409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:44.506935    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr" podUID="3e4c8521-f6ec-4d79-8946-045048bdad39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:04:44.898296 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:44.898261    4553 patch_prober.go:29] interesting pod/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:44.898517 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:44.898325    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" podUID="eedadc43-2c7f-465a-a834-f1871f020e61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:45.521158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:45.517839    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:45.521158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:45.517898    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr" podUID="3e4c8521-f6ec-4d79-8946-045048bdad39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:04:45.581155 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:45.581108    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:45.581451 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:45.581425    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:45.582058 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:45.582040    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:04:46.506792 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:46.506749    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:46.506976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:46.506806    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr" podUID="3e4c8521-f6ec-4d79-8946-045048bdad39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:04:47.506707 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:47.506673    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:04:47.507314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:47.506732    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr" podUID="3e4c8521-f6ec-4d79-8946-045048bdad39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:04:48.991516 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:48.991483    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:48.992859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:48.992835    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:48.994553 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:04:48.994531    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:04:58.202283 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:04:58.202239    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:05:10.120143 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.120109    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"]
Oct 30 22:05:10.125416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.125385    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:10.182742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.182701    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"]
Oct 30 22:05:10.198216 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.198167    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-metrics-certs\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:10.198388 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.198325    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-maas-api-tls\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:10.198388 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.198358    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkdps\" (UniqueName: \"kubernetes.io/projected/8324c7c5-ac7a-439d-9f12-048a609e6ab3-kube-api-access-mkdps\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:10.198480 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.198436    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8324c7c5-ac7a-439d-9f12-048a609e6ab3-openshift-service-ca\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:10.229212 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.229182    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7fabe91d-serving-cert\""
Oct 30 22:05:10.302975 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.302932    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-metrics-certs\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:10.303212 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.303031    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-maas-api-tls\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:10.303212 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.303057    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkdps\" (UniqueName: \"kubernetes.io/projected/8324c7c5-ac7a-439d-9f12-048a609e6ab3-kube-api-access-mkdps\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:10.303212 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.303107    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8324c7c5-ac7a-439d-9f12-048a609e6ab3-openshift-service-ca\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:10.304123 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.304094    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8324c7c5-ac7a-439d-9f12-048a609e6ab3-openshift-service-ca\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:10.305609 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:05:10.305583    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-serving-cert: secret "maas-api-e2e-mt-7fabe91d-serving-cert" not found
Oct 30 22:05:10.305843 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:05:10.305831    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-maas-api-tls podName:8324c7c5-ac7a-439d-9f12-048a609e6ab3 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:10.80579949 +0000 UTC m=+1361.540162514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-maas-api-tls") pod "maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh" (UID: "8324c7c5-ac7a-439d-9f12-048a609e6ab3") : secret "maas-api-e2e-mt-7fabe91d-serving-cert" not found
Oct 30 22:05:10.309423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.309388    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-metrics-certs\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:10.398214 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.389680    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkdps\" (UniqueName: \"kubernetes.io/projected/8324c7c5-ac7a-439d-9f12-048a609e6ab3-kube-api-access-mkdps\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:10.818824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.818741    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-maas-api-tls\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:10.821335 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:10.821311    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-maas-api-tls\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:11.039442 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:11.039404    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:11.426669 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:11.426634    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"]
Oct 30 22:05:11.427151 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:05:11.426959    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8324c7c5_ac7a_439d_9f12_048a609e6ab3.slice/crio-d7ed58c7435a04119653bc3bf9836530ce24054830f571d8f1f172a6a51a5976 WatchSource:0}: Error finding container d7ed58c7435a04119653bc3bf9836530ce24054830f571d8f1f172a6a51a5976: Status 404 returned error can't find the container with id d7ed58c7435a04119653bc3bf9836530ce24054830f571d8f1f172a6a51a5976
Oct 30 22:05:11.429027 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:11.428995    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:05:12.234657 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:12.234623    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh" event={"ID":"8324c7c5-ac7a-439d-9f12-048a609e6ab3","Type":"ContainerStarted","Data":"d7ed58c7435a04119653bc3bf9836530ce24054830f571d8f1f172a6a51a5976"}
Oct 30 22:05:13.243774 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:13.243739    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh" event={"ID":"8324c7c5-ac7a-439d-9f12-048a609e6ab3","Type":"ContainerStarted","Data":"8232d3a1cea90b3d8774a27be51a4794c70314959d53bfdb3138b06520c84fed"}
Oct 30 22:05:13.244346 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:13.243889    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:13.370091 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:13.370014    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh" podStartSLOduration=2.462340239 podStartE2EDuration="3.369963836s" totalImagesPullingTime="907.623597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:13.367604144 +0000 UTC m=+1364.101967176" watchObservedRunningTime="2026-10-30 22:05:13.369963836 +0000 UTC m=+1364.104326867"
Oct 30 22:05:19.288161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:19.287334    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:05:19.585222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:19.574407    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:19.709164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:19.683780    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:19.901968 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:19.901835    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:05:20.218719 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:20.218605    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:20.294169 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:20.240909    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:05:20.294874 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:20.294838    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr" podUID="3e4c8521-f6ec-4d79-8946-045048bdad39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:20.365157 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:20.334876    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:05:20.586222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:20.581570    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 22:05:20.586222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:20.583427    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 22:05:20.963407 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:20.963304    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:21.005163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:20.964385    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:05:21.266444 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:21.266333    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:21.314179 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:21.314124    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 22:05:21.362485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:21.362326    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tdqzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:05:21.362485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:21.362416    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:21.370902 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:21.370867    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-krsdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:05:21.371238 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:21.371206    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:21.395162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:21.371461    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Oct 30 22:05:21.395162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:21.371859    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:05:21.610242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:21.583241    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 22:05:21.829161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:21.800720    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 22:05:21.829161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:21.800788    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:05:21.944358 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:21.944243    4553 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:22.132323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:22.132272    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:22.195972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:22.195876    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 22:05:22.338174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:22.337742    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:05:28.204259 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:05:28.204224    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:05:32.912840 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:32.912802    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"]
Oct 30 22:05:32.918330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:32.918301    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:32.976347 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:32.976312    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"]
Oct 30 22:05:33.022507 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.022473    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-90e0d5\"/\"openshift-service-ca.crt\""
Oct 30 22:05:33.022507 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.022507    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-90e0d5\"/\"default-dockercfg-ft4lg\""
Oct 30 22:05:33.022745 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.022586    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-90e0d5\"/\"distinct-2-w2-90e0d5-kserve-self-signed-certs\""
Oct 30 22:05:33.022844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.022823    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-90e0d5\"/\"kube-root-ca.crt\""
Oct 30 22:05:33.068162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.065685    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-model-cache\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.068162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.065749    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-tls-certs\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.068162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.065894    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-dshm\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.068162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.066022    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8rt8\" (UniqueName: \"kubernetes.io/projected/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-kube-api-access-k8rt8\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.068162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.066075    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-home\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.155405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.155362    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-tmp-dir\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.258392 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.258285    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-home\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.258392 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.258350    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-tmp-dir\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.258646 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.258457    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-model-cache\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.258646 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.258478    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-tls-certs\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.258646 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.258557    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-dshm\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.258646 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.258642    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8rt8\" (UniqueName: \"kubernetes.io/projected/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-kube-api-access-k8rt8\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.258885 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.258863    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-tmp-dir\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.258953 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.258917    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-home\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.258953 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.258935    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-model-cache\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.261004 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.260979    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-dshm\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.261308 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.261285    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-tls-certs\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.373163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.365880    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8rt8\" (UniqueName: \"kubernetes.io/projected/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-kube-api-access-k8rt8\") pod \"distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") " pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.532527 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.532435    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:33.916349 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:33.916314    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"]
Oct 30 22:05:33.917351 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:05:33.917311    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12e366ed_b8f7_4d11_aae0_9f3e4015c8cc.slice/crio-1bf43e3bdb3e0d75562aa8f55bc2a7d367dd94695b2595c3d02539d870a3ece3 WatchSource:0}: Error finding container 1bf43e3bdb3e0d75562aa8f55bc2a7d367dd94695b2595c3d02539d870a3ece3: Status 404 returned error can't find the container with id 1bf43e3bdb3e0d75562aa8f55bc2a7d367dd94695b2595c3d02539d870a3ece3
Oct 30 22:05:34.581394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:34.581354    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v" event={"ID":"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc","Type":"ContainerStarted","Data":"f5db27f72443ee9a4ece8a6991e991d858f5cafae9dfb0234c521875e399cafc"}
Oct 30 22:05:34.581578 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:34.581407    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v" event={"ID":"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc","Type":"ContainerStarted","Data":"1bf43e3bdb3e0d75562aa8f55bc2a7d367dd94695b2595c3d02539d870a3ece3"}
Oct 30 22:05:34.701534 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:34.701482    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v" podStartSLOduration=2.701466623 podStartE2EDuration="2.701466623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:34.69893033 +0000 UTC m=+1385.433293353" watchObservedRunningTime="2026-10-30 22:05:34.701466623 +0000 UTC m=+1385.435829653"
Oct 30 22:05:35.532780 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:35.532743    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:38.909623 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:38.909525    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"]
Oct 30 22:05:38.915028 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:38.914999    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.190300 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.069649    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9ac4a7-openshift-default-dockercfg-89nh7\""
Oct 30 22:05:39.218212 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.072011    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-credential-socket\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.218487 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.218465    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-data\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.245187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.218633    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447fed40-9a8d-4a48-88a3-c9028d7792d5-istiod-ca-cert\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.245187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.237311    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-envoy\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.245187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.237416    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-workload-socket\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.245187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.237464    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-token\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.245187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.237494    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x78qb\" (UniqueName: \"kubernetes.io/projected/447fed40-9a8d-4a48-88a3-c9028d7792d5-kube-api-access-x78qb\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.245187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.237578    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-workload-certs\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.245187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.237612    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-podinfo\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.310871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.310782    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"]
Oct 30 22:05:39.341188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.339444    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-workload-socket\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.341188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.339521    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-token\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.341188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.339553    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x78qb\" (UniqueName: \"kubernetes.io/projected/447fed40-9a8d-4a48-88a3-c9028d7792d5-kube-api-access-x78qb\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.341188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.339627    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-workload-certs\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.341188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.339658    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-podinfo\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.341188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.339826    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-credential-socket\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.341188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.339871    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-data\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.341188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.339970    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447fed40-9a8d-4a48-88a3-c9028d7792d5-istiod-ca-cert\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.341188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.340063    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-envoy\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.341188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.340985    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-workload-certs\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.346096 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.342533    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-data\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.346096 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.344310    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447fed40-9a8d-4a48-88a3-c9028d7792d5-istiod-ca-cert\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.346096 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.345272    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-credential-socket\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.346096 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.346047    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-workload-socket\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.431159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.418021    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x78qb\" (UniqueName: \"kubernetes.io/projected/447fed40-9a8d-4a48-88a3-c9028d7792d5-kube-api-access-x78qb\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.431159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.427984    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-envoy\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.431159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.428976    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-podinfo\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.433015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.432684    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-token\") pod \"e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") " pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:39.567272 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:39.566210    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:40.047039 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:05:40.047011    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447fed40_9a8d_4a48_88a3_c9028d7792d5.slice/crio-48a29b60bcce943f184732509ed8cc4eb980084d74bc96b7ec8d7519acc4a174 WatchSource:0}: Error finding container 48a29b60bcce943f184732509ed8cc4eb980084d74bc96b7ec8d7519acc4a174: Status 404 returned error can't find the container with id 48a29b60bcce943f184732509ed8cc4eb980084d74bc96b7ec8d7519acc4a174
Oct 30 22:05:40.048038 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:40.048011    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"]
Oct 30 22:05:40.049851 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:40.049815    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:05:40.049964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:40.049897    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:05:40.049964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:40.049941    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:05:40.626945 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:40.626914    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s" event={"ID":"447fed40-9a8d-4a48-88a3-c9028d7792d5","Type":"ContainerStarted","Data":"738e868126c7b7b0845d5bb9c136bf5a31aea1c488ec0b80a122019d1781bc21"}
Oct 30 22:05:40.627230 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:40.626967    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s" event={"ID":"447fed40-9a8d-4a48-88a3-c9028d7792d5","Type":"ContainerStarted","Data":"48a29b60bcce943f184732509ed8cc4eb980084d74bc96b7ec8d7519acc4a174"}
Oct 30 22:05:41.566910 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:41.566879    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:41.572810 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:41.572781    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:41.633239 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:41.633209    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:41.634181 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:41.634123    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:05:41.747823 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:41.747772    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s" podStartSLOduration=3.747759246 podStartE2EDuration="3.747759246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:40.849388176 +0000 UTC m=+1391.583751207" watchObservedRunningTime="2026-10-30 22:05:41.747759246 +0000 UTC m=+1392.482122276"
Oct 30 22:05:43.533614 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:43.533568    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:43.546330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:43.546295    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:43.657461 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:43.657433    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:05:44.056758 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.056726    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"]
Oct 30 22:05:44.061438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.061416    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.240727 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.240404    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee288017-2408-4f01-bcff-1a575529a750-istio-token\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.240727 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.240635    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee288017-2408-4f01-bcff-1a575529a750-istio-podinfo\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.240727 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.240705    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-istio-data\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.241049 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.240802    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-credential-socket\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.241049 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.240883    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-istio-envoy\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.241049 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.240961    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96xqg\" (UniqueName: \"kubernetes.io/projected/ee288017-2408-4f01-bcff-1a575529a750-kube-api-access-96xqg\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.241049 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.240991    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-workload-socket\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.285551 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.241066    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-workload-certs\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.316488 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.296805    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-da0f3d-openshift-default-dockercfg-5ndhr\""
Oct 30 22:05:44.316488 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.303419    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee288017-2408-4f01-bcff-1a575529a750-istiod-ca-cert\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.346456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.346421    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"]
Oct 30 22:05:44.423273 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.422993    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96xqg\" (UniqueName: \"kubernetes.io/projected/ee288017-2408-4f01-bcff-1a575529a750-kube-api-access-96xqg\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.423273 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.423043    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-workload-socket\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.423273 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.423074    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-workload-certs\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.423273 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.423156    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee288017-2408-4f01-bcff-1a575529a750-istiod-ca-cert\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.423273 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.423217    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee288017-2408-4f01-bcff-1a575529a750-istio-token\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.423825 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.423380    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee288017-2408-4f01-bcff-1a575529a750-istio-podinfo\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.423825 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.423443    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-istio-data\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.423825 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.423498    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-credential-socket\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.423825 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.423573    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-istio-envoy\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.458064 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.448443    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-istio-envoy\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.458064 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.449260    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-workload-socket\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.458064 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.449518    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-workload-certs\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.458064 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.450269    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee288017-2408-4f01-bcff-1a575529a750-istiod-ca-cert\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.458064 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.454648    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee288017-2408-4f01-bcff-1a575529a750-istio-podinfo\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.458064 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.454963    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-istio-data\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.458064 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.455309    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-credential-socket\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.542158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.523948    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96xqg\" (UniqueName: \"kubernetes.io/projected/ee288017-2408-4f01-bcff-1a575529a750-kube-api-access-96xqg\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.542158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.524377    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee288017-2408-4f01-bcff-1a575529a750-istio-token\") pod \"e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") " pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:44.676873 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:44.676836    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:45.110782 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:45.110751    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"]
Oct 30 22:05:45.149337 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:05:45.149307    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee288017_2408_4f01_bcff_1a575529a750.slice/crio-6ecf1787b92ae9948623e72741310a67145c4aa1101e7de6120faf4825d24972 WatchSource:0}: Error finding container 6ecf1787b92ae9948623e72741310a67145c4aa1101e7de6120faf4825d24972: Status 404 returned error can't find the container with id 6ecf1787b92ae9948623e72741310a67145c4aa1101e7de6120faf4825d24972
Oct 30 22:05:45.151780 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:45.151744    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:05:45.151880 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:45.151814    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:05:45.151880 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:45.151844    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:05:45.660874 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:45.660827    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4" event={"ID":"ee288017-2408-4f01-bcff-1a575529a750","Type":"ContainerStarted","Data":"e7b06d9e6b79b59e93944855db84920ce7d761854c4fddc7e606f1143ec14b56"}
Oct 30 22:05:45.661413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:45.660898    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4" event={"ID":"ee288017-2408-4f01-bcff-1a575529a750","Type":"ContainerStarted","Data":"6ecf1787b92ae9948623e72741310a67145c4aa1101e7de6120faf4825d24972"}
Oct 30 22:05:45.677591 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:45.677559    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:45.781965 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:45.781917    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4" podStartSLOduration=1.7819014819999999 podStartE2EDuration="1.781901482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:45.779421465 +0000 UTC m=+1396.513784497" watchObservedRunningTime="2026-10-30 22:05:45.781901482 +0000 UTC m=+1396.516264512"
Oct 30 22:05:46.682815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:46.682779    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:47.672661 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:47.672632    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:47.673666 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:47.673643    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:05:58.025604 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.025550    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"]
Oct 30 22:05:58.047160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.046499    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"
Oct 30 22:05:58.104170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.077380    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"]
Oct 30 22:05:58.135173 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.134013    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9ac4a7-dockercfg-zctdx\""
Oct 30 22:05:58.135173 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.134211    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9ac4a7\""
Oct 30 22:05:58.202173 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.199433    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fd1f7f-a04d-4edc-8e59-0887fa39176a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6\" (UID: \"44fd1f7f-a04d-4edc-8e59-0887fa39176a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"
Oct 30 22:05:58.202173 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.199665    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q2ds\" (UniqueName: \"kubernetes.io/projected/44fd1f7f-a04d-4edc-8e59-0887fa39176a-kube-api-access-8q2ds\") pod \"payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6\" (UID: \"44fd1f7f-a04d-4edc-8e59-0887fa39176a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"
Oct 30 22:05:58.209848 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.209806    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"]
Oct 30 22:05:58.380244 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.328821    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q2ds\" (UniqueName: \"kubernetes.io/projected/44fd1f7f-a04d-4edc-8e59-0887fa39176a-kube-api-access-8q2ds\") pod \"payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6\" (UID: \"44fd1f7f-a04d-4edc-8e59-0887fa39176a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"
Oct 30 22:05:58.380884 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.380830    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fd1f7f-a04d-4edc-8e59-0887fa39176a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6\" (UID: \"44fd1f7f-a04d-4edc-8e59-0887fa39176a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"
Oct 30 22:05:58.382096 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.382068    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fd1f7f-a04d-4edc-8e59-0887fa39176a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6\" (UID: \"44fd1f7f-a04d-4edc-8e59-0887fa39176a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"
Oct 30 22:05:58.382437 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.382417    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"
Oct 30 22:05:58.400333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.400285    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"]
Oct 30 22:05:58.400695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.400541    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q2ds\" (UniqueName: \"kubernetes.io/projected/44fd1f7f-a04d-4edc-8e59-0887fa39176a-kube-api-access-8q2ds\") pod \"payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6\" (UID: \"44fd1f7f-a04d-4edc-8e59-0887fa39176a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"
Oct 30 22:05:58.433961 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:05:58.433916    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:05:58.465407 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.465372    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"
Oct 30 22:05:58.482319 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.482288    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a70e922-07d2-475f-861d-a0a2f54c588f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v\" (UID: \"5a70e922-07d2-475f-861d-a0a2f54c588f\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"
Oct 30 22:05:58.482544 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.482524    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjhgk\" (UniqueName: \"kubernetes.io/projected/5a70e922-07d2-475f-861d-a0a2f54c588f-kube-api-access-wjhgk\") pod \"payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v\" (UID: \"5a70e922-07d2-475f-861d-a0a2f54c588f\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"
Oct 30 22:05:58.584513 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.584475    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a70e922-07d2-475f-861d-a0a2f54c588f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v\" (UID: \"5a70e922-07d2-475f-861d-a0a2f54c588f\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"
Oct 30 22:05:58.584732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.584713    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjhgk\" (UniqueName: \"kubernetes.io/projected/5a70e922-07d2-475f-861d-a0a2f54c588f-kube-api-access-wjhgk\") pod \"payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v\" (UID: \"5a70e922-07d2-475f-861d-a0a2f54c588f\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"
Oct 30 22:05:58.585444 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.585417    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a70e922-07d2-475f-861d-a0a2f54c588f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v\" (UID: \"5a70e922-07d2-475f-861d-a0a2f54c588f\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"
Oct 30 22:05:58.643941 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.643855    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjhgk\" (UniqueName: \"kubernetes.io/projected/5a70e922-07d2-475f-861d-a0a2f54c588f-kube-api-access-wjhgk\") pod \"payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v\" (UID: \"5a70e922-07d2-475f-861d-a0a2f54c588f\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"
Oct 30 22:05:58.723323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.723287    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"
Oct 30 22:05:58.848903 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.848871    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"]
Oct 30 22:05:58.852543 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:05:58.852500    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fd1f7f_a04d_4edc_8e59_0887fa39176a.slice/crio-0fc95febf77d670589246f9913d82832cd173f1423a3adc3d160b9d5fe63a585 WatchSource:0}: Error finding container 0fc95febf77d670589246f9913d82832cd173f1423a3adc3d160b9d5fe63a585: Status 404 returned error can't find the container with id 0fc95febf77d670589246f9913d82832cd173f1423a3adc3d160b9d5fe63a585
Oct 30 22:05:58.929979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.929946    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6" event={"ID":"44fd1f7f-a04d-4edc-8e59-0887fa39176a","Type":"ContainerStarted","Data":"002f2783f30653728e25bc3a39decfc3485c82702090142fa4c0ff08bf569cc7"}
Oct 30 22:05:58.930150 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.930004    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6" event={"ID":"44fd1f7f-a04d-4edc-8e59-0887fa39176a","Type":"ContainerStarted","Data":"0fc95febf77d670589246f9913d82832cd173f1423a3adc3d160b9d5fe63a585"}
Oct 30 22:05:58.930150 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:58.930046    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"
Oct 30 22:05:59.046410 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:59.046357    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6" podStartSLOduration=2.046341144 podStartE2EDuration="2.046341144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:59.044229632 +0000 UTC m=+1409.778592663" watchObservedRunningTime="2026-10-30 22:05:59.046341144 +0000 UTC m=+1409.780704192"
Oct 30 22:05:59.102514 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:05:59.102481    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a70e922_07d2_475f_861d_a0a2f54c588f.slice/crio-5198c4ac7ddbb3ce1f3d7e33e8a1ee867c90243161d38675eb7d430957ecdf4c WatchSource:0}: Error finding container 5198c4ac7ddbb3ce1f3d7e33e8a1ee867c90243161d38675eb7d430957ecdf4c: Status 404 returned error can't find the container with id 5198c4ac7ddbb3ce1f3d7e33e8a1ee867c90243161d38675eb7d430957ecdf4c
Oct 30 22:05:59.102758 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:59.102736    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"]
Oct 30 22:05:59.938514 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:59.938469    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v" event={"ID":"5a70e922-07d2-475f-861d-a0a2f54c588f","Type":"ContainerStarted","Data":"1337529090f27bf07d1a1eb0bf70aea3b511e14c4cb8cbdeb306a3eb1da708f1"}
Oct 30 22:05:59.938722 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:59.938533    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"
Oct 30 22:05:59.938722 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:05:59.938552    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v" event={"ID":"5a70e922-07d2-475f-861d-a0a2f54c588f","Type":"ContainerStarted","Data":"5198c4ac7ddbb3ce1f3d7e33e8a1ee867c90243161d38675eb7d430957ecdf4c"}
Oct 30 22:06:00.055498 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:00.055444    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v" podStartSLOduration=2.05542847 podStartE2EDuration="2.05542847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:00.053877265 +0000 UTC m=+1410.788240295" watchObservedRunningTime="2026-10-30 22:06:00.05542847 +0000 UTC m=+1410.789791502"
Oct 30 22:06:03.175159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:03.174955    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"]
Oct 30 22:06:03.221163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:03.180447    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"
Oct 30 22:06:03.330107 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:03.330061    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"]
Oct 30 22:06:03.343581 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:03.336542    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-da0f3d\""
Oct 30 22:06:03.343581 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:03.336698    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-da0f3d-dockercfg-xgrxt\""
Oct 30 22:06:03.468790 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:03.467564    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8csh\" (UniqueName: \"kubernetes.io/projected/767d4fe8-a9f3-4627-bd29-68732f291535-kube-api-access-f8csh\") pod \"payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l\" (UID: \"767d4fe8-a9f3-4627-bd29-68732f291535\") " pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"
Oct 30 22:06:03.468790 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:03.467708    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767d4fe8-a9f3-4627-bd29-68732f291535-plugins-config-volume\") pod \"payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l\" (UID: \"767d4fe8-a9f3-4627-bd29-68732f291535\") " pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"
Oct 30 22:06:03.569305 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:03.569270    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8csh\" (UniqueName: \"kubernetes.io/projected/767d4fe8-a9f3-4627-bd29-68732f291535-kube-api-access-f8csh\") pod \"payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l\" (UID: \"767d4fe8-a9f3-4627-bd29-68732f291535\") " pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"
Oct 30 22:06:03.569480 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:03.569365    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767d4fe8-a9f3-4627-bd29-68732f291535-plugins-config-volume\") pod \"payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l\" (UID: \"767d4fe8-a9f3-4627-bd29-68732f291535\") " pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"
Oct 30 22:06:03.570114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:03.570093    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767d4fe8-a9f3-4627-bd29-68732f291535-plugins-config-volume\") pod \"payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l\" (UID: \"767d4fe8-a9f3-4627-bd29-68732f291535\") " pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"
Oct 30 22:06:03.628154 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:03.628104    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8csh\" (UniqueName: \"kubernetes.io/projected/767d4fe8-a9f3-4627-bd29-68732f291535-kube-api-access-f8csh\") pod \"payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l\" (UID: \"767d4fe8-a9f3-4627-bd29-68732f291535\") " pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"
Oct 30 22:06:03.649249 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:03.649222    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"
Oct 30 22:06:04.024098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:04.024066    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"]
Oct 30 22:06:04.025366 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:06:04.025342    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767d4fe8_a9f3_4627_bd29_68732f291535.slice/crio-418bc8cd1b180d43cb75656868ffd512cb36c36f9c4f6e9d78304460c63b944c WatchSource:0}: Error finding container 418bc8cd1b180d43cb75656868ffd512cb36c36f9c4f6e9d78304460c63b944c: Status 404 returned error can't find the container with id 418bc8cd1b180d43cb75656868ffd512cb36c36f9c4f6e9d78304460c63b944c
Oct 30 22:06:04.062871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:04.062831    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l" event={"ID":"767d4fe8-a9f3-4627-bd29-68732f291535","Type":"ContainerStarted","Data":"418bc8cd1b180d43cb75656868ffd512cb36c36f9c4f6e9d78304460c63b944c"}
Oct 30 22:06:05.070397 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:05.070363    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l" event={"ID":"767d4fe8-a9f3-4627-bd29-68732f291535","Type":"ContainerStarted","Data":"11e14ef318a24a3b6782c9c0351393d0428204c64684d72bb55d974da1a44eab"}
Oct 30 22:06:05.070870 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:05.070502    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"
Oct 30 22:06:05.216019 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:05.215954    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l" podStartSLOduration=2.215939619 podStartE2EDuration="2.215939619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:05.213009066 +0000 UTC m=+1415.947372098" watchObservedRunningTime="2026-10-30 22:06:05.215939619 +0000 UTC m=+1415.950302712"
Oct 30 22:06:09.942296 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:09.942263    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"
Oct 30 22:06:10.948861 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:10.948835    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"
Oct 30 22:06:16.080614 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:16.080586    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"
Oct 30 22:06:27.681479 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:27.681443    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"]
Oct 30 22:06:27.685986 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:27.685964    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:27.743674 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:27.743640    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"]
Oct 30 22:06:27.770955 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:27.770904    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-maas-api-tls\") pod \"maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:27.770955 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:27.770958    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86zpb\" (UniqueName: \"kubernetes.io/projected/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-kube-api-access-86zpb\") pod \"maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:27.771231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:27.771062    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-metrics-certs\") pod \"maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:27.771287 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:27.771275    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-openshift-service-ca\") pod \"maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:27.788822 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:27.788785    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9ac4a7-serving-cert\""
Oct 30 22:06:27.872875 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:27.872837    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-openshift-service-ca\") pod \"maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:27.873068 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:27.873033    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-maas-api-tls\") pod \"maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:27.873068 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:27.873058    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86zpb\" (UniqueName: \"kubernetes.io/projected/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-kube-api-access-86zpb\") pod \"maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:27.873260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:27.873084    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-metrics-certs\") pod \"maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:27.873260 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:06:27.873225    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-serving-cert: secret "maas-api-e2e-api-a-9ac4a7-serving-cert" not found
Oct 30 22:06:27.873378 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:06:27.873322    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-maas-api-tls podName:9e4d1744-6b44-46ae-8e36-0f3584c10ed6 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:28.37330121 +0000 UTC m=+1439.107664244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-maas-api-tls") pod "maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm" (UID: "9e4d1744-6b44-46ae-8e36-0f3584c10ed6") : secret "maas-api-e2e-api-a-9ac4a7-serving-cert" not found
Oct 30 22:06:27.873576 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:27.873555    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-openshift-service-ca\") pod \"maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:27.875950 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:27.875924    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-metrics-certs\") pod \"maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:27.955223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:27.954173    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86zpb\" (UniqueName: \"kubernetes.io/projected/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-kube-api-access-86zpb\") pod \"maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:28.301241 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:06:28.299559    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:06:28.406960 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:28.406919    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-maas-api-tls\") pod \"maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:28.409681 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:28.409657    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-maas-api-tls\") pod \"maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:28.600538 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:28.600501    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:28.987824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:28.987792    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"]
Oct 30 22:06:28.988345 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:06:28.988049    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4d1744_6b44_46ae_8e36_0f3584c10ed6.slice/crio-070bbd2ff6d9ffcb8463497dafb9246a556322ffa8b175ed1cdb9ec05cf66ebb WatchSource:0}: Error finding container 070bbd2ff6d9ffcb8463497dafb9246a556322ffa8b175ed1cdb9ec05cf66ebb: Status 404 returned error can't find the container with id 070bbd2ff6d9ffcb8463497dafb9246a556322ffa8b175ed1cdb9ec05cf66ebb
Oct 30 22:06:29.314421 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:29.314334    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm" event={"ID":"9e4d1744-6b44-46ae-8e36-0f3584c10ed6","Type":"ContainerStarted","Data":"070bbd2ff6d9ffcb8463497dafb9246a556322ffa8b175ed1cdb9ec05cf66ebb"}
Oct 30 22:06:30.322767 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:30.322661    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm" event={"ID":"9e4d1744-6b44-46ae-8e36-0f3584c10ed6","Type":"ContainerStarted","Data":"c427a2387ea342a70d8e017e05a5751dfd3d78d886ea0d8b954413cdb1dbfd32"}
Oct 30 22:06:30.323165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:30.322766    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:30.439364 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:30.439310    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm" podStartSLOduration=2.47216125 podStartE2EDuration="3.439290055s" totalImagesPullingTime="967.128805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:30.438410037 +0000 UTC m=+1441.172773069" watchObservedRunningTime="2026-10-30 22:06:30.439290055 +0000 UTC m=+1441.173653086"
Oct 30 22:06:36.335992 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:36.335963    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:06:52.497443 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:52.497355    4553 generic.go:356] "Generic (PLEG): container finished" podID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerID="738e868126c7b7b0845d5bb9c136bf5a31aea1c488ec0b80a122019d1781bc21" exitCode=0
Oct 30 22:06:52.497443 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:52.497433    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s" event={"ID":"447fed40-9a8d-4a48-88a3-c9028d7792d5","Type":"ContainerDied","Data":"738e868126c7b7b0845d5bb9c136bf5a31aea1c488ec0b80a122019d1781bc21"}
Oct 30 22:06:52.497923 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:52.497894    4553 scope.go:122] "RemoveContainer" containerID="738e868126c7b7b0845d5bb9c136bf5a31aea1c488ec0b80a122019d1781bc21"
Oct 30 22:06:52.499643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:52.499612    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:06:52.499748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:52.499686    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:06:52.499748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:52.499720    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:06:53.506048 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:53.506015    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s" event={"ID":"447fed40-9a8d-4a48-88a3-c9028d7792d5","Type":"ContainerStarted","Data":"0be1c22599e8a6fb471786b544b570e4ab720231fe805d7f79a5ecd126ee4666"}
Oct 30 22:06:53.567183 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:53.567111    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:06:53.568227 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:53.568196    4553 patch_prober.go:29] interesting pod/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:06:53.568344 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:53.568259    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s" podUID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 22:06:54.567397 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:54.567366    4553 patch_prober.go:29] interesting pod/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:06:54.567833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:54.567420    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s" podUID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 22:06:55.567663 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:55.567630    4553 patch_prober.go:29] interesting pod/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:06:55.568085 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:55.567686    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s" podUID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 22:06:56.567302 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:56.567264    4553 patch_prober.go:29] interesting pod/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:06:56.567490 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:56.567329    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s" podUID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 22:06:56.633310 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:56.633279    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:06:57.567608 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:57.567577    4553 patch_prober.go:29] interesting pod/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:06:57.567804 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:57.567627    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s" podUID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 22:06:58.205531 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:06:58.205496    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:06:58.567404 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:58.567310    4553 patch_prober.go:29] interesting pod/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Oct 30 22:06:58.567404 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:58.567383    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s" podUID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Oct 30 22:06:59.572088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:59.572056    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:06:59.573298 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:06:59.573278    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:07:04.562319 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.562278    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"]
Oct 30 22:07:04.568704 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.568682    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.599222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.599184    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"]
Oct 30 22:07:04.603604 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.603581    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.651094 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.651063    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"]
Oct 30 22:07:04.680345 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.680313    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.680524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.680362    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.680524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.680385    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.680524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.680485    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.680524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.680517    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.680658 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.680543    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.680658 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.680624    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.680658 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.680642    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494ed239-bcbe-4ba4-a79d-a46b23d62456-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.680749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.680728    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.680782 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.680760    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.680782 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.680777    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zjcl\" (UniqueName: \"kubernetes.io/projected/494ed239-bcbe-4ba4-a79d-a46b23d62456-kube-api-access-9zjcl\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.680875 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.680861    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.680934 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.680920    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.681041 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.681008    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.681041 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.681040    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffn2f\" (UniqueName: \"kubernetes.io/projected/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-kube-api-access-ffn2f\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.681315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.681079    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.681315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.681172    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.681424 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.681327    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.710415 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.710382    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"]
Oct 30 22:07:04.783270 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783231    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.783456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783317    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.783456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783354    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffn2f\" (UniqueName: \"kubernetes.io/projected/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-kube-api-access-ffn2f\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.783456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783391    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.783456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783446    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.783674 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783509    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.783674 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783619    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.783787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783671    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.783787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783683    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.783787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783704    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.783787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783782    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.784008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783816    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.784008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783854    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.784008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783886    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.784008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783912    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494ed239-bcbe-4ba4-a79d-a46b23d62456-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.784008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.783996    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.784008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.784004    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.784360 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.784060    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.784360 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.784092    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zjcl\" (UniqueName: \"kubernetes.io/projected/494ed239-bcbe-4ba4-a79d-a46b23d62456-kube-api-access-9zjcl\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.784360 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.784152    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.784360 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.784336    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.784593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.784512    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.784593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.784570    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.784815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.784792    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.785036 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.785015    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.785590 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.785566    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.785905 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.785628    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.787023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.786993    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494ed239-bcbe-4ba4-a79d-a46b23d62456-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.787587 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.787558    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.788787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.788748    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.788878 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.788832    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.789218 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.789198    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.843758 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.843678    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zjcl\" (UniqueName: \"kubernetes.io/projected/494ed239-bcbe-4ba4-a79d-a46b23d62456-kube-api-access-9zjcl\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.844469 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.844424    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:04.844646 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.844624    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffn2f\" (UniqueName: \"kubernetes.io/projected/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-kube-api-access-ffn2f\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.851682 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.851651    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.883430 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.883399    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:04.917836 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:04.917798    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:05.322278 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:07:05.322241    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494ed239_bcbe_4ba4_a79d_a46b23d62456.slice/crio-a6bbe0cf2764362d153c8d62121f152561f8490aa0adbb475ea0a498afe96a62 WatchSource:0}: Error finding container a6bbe0cf2764362d153c8d62121f152561f8490aa0adbb475ea0a498afe96a62: Status 404 returned error can't find the container with id a6bbe0cf2764362d153c8d62121f152561f8490aa0adbb475ea0a498afe96a62
Oct 30 22:07:05.323476 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.323453    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"]
Oct 30 22:07:05.325492 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.325458    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:07:05.325594 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.325535    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:07:05.325594 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.325564    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:07:05.327187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.327163    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"]
Oct 30 22:07:05.367766 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:07:05.367737    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9bfbd5_1f38_4154_b3a3_d7ad054fe4e5.slice/crio-5199d3687cee2889c9e9b1862ff6ec7a87949043f5442b4a83bfea4fe3d98012 WatchSource:0}: Error finding container 5199d3687cee2889c9e9b1862ff6ec7a87949043f5442b4a83bfea4fe3d98012: Status 404 returned error can't find the container with id 5199d3687cee2889c9e9b1862ff6ec7a87949043f5442b4a83bfea4fe3d98012
Oct 30 22:07:05.370788 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.370754    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:07:05.370873 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.370823    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:07:05.370873 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.370853    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:07:05.615337 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.615297    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq" event={"ID":"494ed239-bcbe-4ba4-a79d-a46b23d62456","Type":"ContainerStarted","Data":"bc6710ec824a290d29c53fe6b902a583152eaa543339acfd91b1151e43ee1089"}
Oct 30 22:07:05.615761 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.615365    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq" event={"ID":"494ed239-bcbe-4ba4-a79d-a46b23d62456","Type":"ContainerStarted","Data":"a6bbe0cf2764362d153c8d62121f152561f8490aa0adbb475ea0a498afe96a62"}
Oct 30 22:07:05.618164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.618107    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p" event={"ID":"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5","Type":"ContainerStarted","Data":"155a54961fd6e67e31e1f4564fc25d8d1106fec67decba7d28c8c4c536ac7f58"}
Oct 30 22:07:05.618363 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.618216    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p" event={"ID":"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5","Type":"ContainerStarted","Data":"5199d3687cee2889c9e9b1862ff6ec7a87949043f5442b4a83bfea4fe3d98012"}
Oct 30 22:07:05.734158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.733776    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq" podStartSLOduration=1.733756927 podStartE2EDuration="1.733756927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:05.733109279 +0000 UTC m=+1476.467472316" watchObservedRunningTime="2026-10-30 22:07:05.733756927 +0000 UTC m=+1476.468119958"
Oct 30 22:07:05.852034 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.851983    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p" podStartSLOduration=1.851967076 podStartE2EDuration="1.851967076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:05.849796157 +0000 UTC m=+1476.584159188" watchObservedRunningTime="2026-10-30 22:07:05.851967076 +0000 UTC m=+1476.586330107"
Oct 30 22:07:05.884497 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.884409    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:05.917884 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:05.917854    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:06.885370 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:06.885330    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 22:07:06.885769 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:06.885423    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p" podUID="5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 22:07:06.931168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:06.926008    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:07:06.931168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:06.926072    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq" podUID="494ed239-bcbe-4ba4-a79d-a46b23d62456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 22:07:07.884411 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:07.884373    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 22:07:07.884754 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:07.884722    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p" podUID="5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 22:07:07.933161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:07.932275    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:07:07.933161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:07.932390    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq" podUID="494ed239-bcbe-4ba4-a79d-a46b23d62456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 22:07:08.884525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:08.884450    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 22:07:08.884803 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:08.884777    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p" podUID="5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 22:07:08.918742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:08.918609    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:07:08.918742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:08.918682    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq" podUID="494ed239-bcbe-4ba4-a79d-a46b23d62456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 22:07:09.883811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:09.883776    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 22:07:09.884291 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:09.883841    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p" podUID="5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 22:07:09.929271 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:09.929232    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:07:09.929526 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:09.929497    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq" podUID="494ed239-bcbe-4ba4-a79d-a46b23d62456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 22:07:10.891741 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:10.890713    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 22:07:10.891741 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:10.890797    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p" podUID="5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 22:07:10.920524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:10.920487    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:07:10.920815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:10.920785    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq" podUID="494ed239-bcbe-4ba4-a79d-a46b23d62456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 22:07:11.890163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:11.889939    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 22:07:11.890163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:11.890004    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p" podUID="5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 22:07:11.934168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:11.933731    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:07:11.934168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:11.933884    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq" podUID="494ed239-bcbe-4ba4-a79d-a46b23d62456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 22:07:12.883800 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:12.883762    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 22:07:12.883994 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:12.883821    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p" podUID="5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 22:07:12.925171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:12.920530    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 22:07:12.925171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:12.920703    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq" podUID="494ed239-bcbe-4ba4-a79d-a46b23d62456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 22:07:14.307128 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:14.306181    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:14.307990 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:14.307965    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:14.317159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:14.313405    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:07:14.457548 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:14.457445    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:14.458853 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:14.458791    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:14.460041 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:14.459986    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:07:16.994809 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:16.994777    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"]
Oct 30 22:07:17.003672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.003649    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"
Oct 30 22:07:17.054913 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.054880    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"]
Oct 30 22:07:17.101220 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.101188    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d73ad7-dockercfg-6p46k\""
Oct 30 22:07:17.101446 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.101270    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d73ad7\""
Oct 30 22:07:17.153350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.134484    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc64ce9a-ba29-44aa-a770-f98d29183190-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx\" (UID: \"cc64ce9a-ba29-44aa-a770-f98d29183190\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"
Oct 30 22:07:17.153350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.134558    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfdgn\" (UniqueName: \"kubernetes.io/projected/cc64ce9a-ba29-44aa-a770-f98d29183190-kube-api-access-mfdgn\") pod \"payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx\" (UID: \"cc64ce9a-ba29-44aa-a770-f98d29183190\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"
Oct 30 22:07:17.242073 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.236163    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfdgn\" (UniqueName: \"kubernetes.io/projected/cc64ce9a-ba29-44aa-a770-f98d29183190-kube-api-access-mfdgn\") pod \"payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx\" (UID: \"cc64ce9a-ba29-44aa-a770-f98d29183190\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"
Oct 30 22:07:17.242777 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.242740    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc64ce9a-ba29-44aa-a770-f98d29183190-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx\" (UID: \"cc64ce9a-ba29-44aa-a770-f98d29183190\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"
Oct 30 22:07:17.243973 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.243948    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc64ce9a-ba29-44aa-a770-f98d29183190-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx\" (UID: \"cc64ce9a-ba29-44aa-a770-f98d29183190\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"
Oct 30 22:07:17.268480 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.268392    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"]
Oct 30 22:07:17.288808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.288781    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"
Oct 30 22:07:17.345312 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.304895    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"]
Oct 30 22:07:17.464998 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.464967    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp\" (UID: \"f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd\") " pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"
Oct 30 22:07:17.467056 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.465662    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpmnh\" (UniqueName: \"kubernetes.io/projected/f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd-kube-api-access-rpmnh\") pod \"payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp\" (UID: \"f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd\") " pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"
Oct 30 22:07:17.467056 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.465865    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfdgn\" (UniqueName: \"kubernetes.io/projected/cc64ce9a-ba29-44aa-a770-f98d29183190-kube-api-access-mfdgn\") pod \"payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx\" (UID: \"cc64ce9a-ba29-44aa-a770-f98d29183190\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"
Oct 30 22:07:17.568521 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.568437    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp\" (UID: \"f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd\") " pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"
Oct 30 22:07:17.568688 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.568531    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpmnh\" (UniqueName: \"kubernetes.io/projected/f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd-kube-api-access-rpmnh\") pod \"payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp\" (UID: \"f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd\") " pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"
Oct 30 22:07:17.569250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.569228    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp\" (UID: \"f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd\") " pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"
Oct 30 22:07:17.622060 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.622017    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"
Oct 30 22:07:17.626733 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.626701    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpmnh\" (UniqueName: \"kubernetes.io/projected/f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd-kube-api-access-rpmnh\") pod \"payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp\" (UID: \"f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd\") " pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"
Oct 30 22:07:17.769713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:17.769644    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"
Oct 30 22:07:18.005223 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:07:18.005183    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc64ce9a_ba29_44aa_a770_f98d29183190.slice/crio-7c52842ac67e11e90b807548cb8922f27cf37080183ac02c48c2ae8e29fa5611 WatchSource:0}: Error finding container 7c52842ac67e11e90b807548cb8922f27cf37080183ac02c48c2ae8e29fa5611: Status 404 returned error can't find the container with id 7c52842ac67e11e90b807548cb8922f27cf37080183ac02c48c2ae8e29fa5611
Oct 30 22:07:18.006608 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:18.006576    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"]
Oct 30 22:07:18.014200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:18.014172    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx" event={"ID":"cc64ce9a-ba29-44aa-a770-f98d29183190","Type":"ContainerStarted","Data":"7c52842ac67e11e90b807548cb8922f27cf37080183ac02c48c2ae8e29fa5611"}
Oct 30 22:07:18.197700 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:18.197663    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"]
Oct 30 22:07:18.714374 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:18.714281    4553 patch_prober.go:29] interesting pod/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:07:18.714374 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:18.714360    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4" podUID="ee288017-2408-4f01-bcff-1a575529a750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:07:19.027077 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:19.026988    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp" event={"ID":"f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd","Type":"ContainerStarted","Data":"0773f0d07c503dc884cbb01ecadd309ebc074d6bd503a36b40fef3910983d8da"}
Oct 30 22:07:19.027644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:19.027092    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"
Oct 30 22:07:19.027644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:19.027169    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp" event={"ID":"f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd","Type":"ContainerStarted","Data":"0cb3117e2c0ef43dc2e693539f5b87b76ed59ff1bcab4bfd9d453cc693ca757e"}
Oct 30 22:07:19.029889 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:19.029860    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx" event={"ID":"cc64ce9a-ba29-44aa-a770-f98d29183190","Type":"ContainerStarted","Data":"89664c8b57a0ef4ecebcf59deddd357cd8fa41092ee0c90b01737d2bb7260014"}
Oct 30 22:07:19.030048 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:19.029917    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"
Oct 30 22:07:19.165406 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:19.147820    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp" podStartSLOduration=2.147800341 podStartE2EDuration="2.147800341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:19.147393795 +0000 UTC m=+1489.881756827" watchObservedRunningTime="2026-10-30 22:07:19.147800341 +0000 UTC m=+1489.882163373"
Oct 30 22:07:19.274164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:19.271520    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx" podStartSLOduration=3.27150232 podStartE2EDuration="3.27150232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:19.26660836 +0000 UTC m=+1490.000971393" watchObservedRunningTime="2026-10-30 22:07:19.27150232 +0000 UTC m=+1490.005865351"
Oct 30 22:07:23.074277 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:23.074240    4553 generic.go:356] "Generic (PLEG): container finished" podID="ee288017-2408-4f01-bcff-1a575529a750" containerID="e7b06d9e6b79b59e93944855db84920ce7d761854c4fddc7e606f1143ec14b56" exitCode=0
Oct 30 22:07:23.074837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:23.074321    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4" event={"ID":"ee288017-2408-4f01-bcff-1a575529a750","Type":"ContainerDied","Data":"e7b06d9e6b79b59e93944855db84920ce7d761854c4fddc7e606f1143ec14b56"}
Oct 30 22:07:23.074934 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:23.074915    4553 scope.go:122] "RemoveContainer" containerID="e7b06d9e6b79b59e93944855db84920ce7d761854c4fddc7e606f1143ec14b56"
Oct 30 22:07:23.076734 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:23.076705    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:07:23.076848 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:23.076786    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:07:23.076848 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:23.076822    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:07:24.082871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:24.082840    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4" event={"ID":"ee288017-2408-4f01-bcff-1a575529a750","Type":"ContainerStarted","Data":"490247fbebf06adffec50addafeaf0cbf6f0c0010792b0d2d4394d87554b1b7f"}
Oct 30 22:07:24.677514 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:24.677477    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:07:24.677727 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:24.677707    4553 patch_prober.go:29] interesting pod/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:07:24.677773 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:24.677751    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4" podUID="ee288017-2408-4f01-bcff-1a575529a750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 22:07:25.677461 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:25.677422    4553 patch_prober.go:29] interesting pod/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:07:25.677894 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:25.677490    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4" podUID="ee288017-2408-4f01-bcff-1a575529a750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 22:07:26.677595 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:26.677554    4553 patch_prober.go:29] interesting pod/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:07:26.678101 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:26.677627    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4" podUID="ee288017-2408-4f01-bcff-1a575529a750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 22:07:27.678244 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:27.678211    4553 patch_prober.go:29] interesting pod/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:07:27.678683 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:27.678265    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4" podUID="ee288017-2408-4f01-bcff-1a575529a750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 22:07:28.232549 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:07:28.232471    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:07:28.678112 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:28.678072    4553 patch_prober.go:29] interesting pod/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 30 22:07:28.678330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:28.678183    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4" podUID="ee288017-2408-4f01-bcff-1a575529a750" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 30 22:07:29.652401 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:29.652371    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:07:29.653049 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:29.653025    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:07:29.700997 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:29.700957    4553 scope.go:122] "RemoveContainer" containerID="90d2f9f62877017b2a62afdefd0464f3d58a8c80a0949569290df2665d10fd8c"
Oct 30 22:07:29.712289 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:29.712257    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:07:29.713672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:29.713635    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:07:29.714750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:29.714722    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:07:30.053441 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:30.053360    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"
Oct 30 22:07:30.055827 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:30.055800    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"
Oct 30 22:07:39.399562 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:39.399525    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"]
Oct 30 22:07:39.516329 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:39.516291    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"
Oct 30 22:07:39.696295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:39.695916    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-855wx\" (UniqueName: \"kubernetes.io/projected/e5b5b84d-ccf6-4683-af90-ba5aef6c821e-kube-api-access-855wx\") pod \"payload-processing-e2e-api-b-02a8fa-6987989665-drfl8\" (UID: \"e5b5b84d-ccf6-4683-af90-ba5aef6c821e\") " pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"
Oct 30 22:07:39.745918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:39.696645    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b5b84d-ccf6-4683-af90-ba5aef6c821e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-02a8fa-6987989665-drfl8\" (UID: \"e5b5b84d-ccf6-4683-af90-ba5aef6c821e\") " pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"
Oct 30 22:07:39.745918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:39.706335    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-02a8fa-dockercfg-cb8cq\""
Oct 30 22:07:39.745918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:39.714178    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-02a8fa\""
Oct 30 22:07:39.814172 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:39.802226    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-855wx\" (UniqueName: \"kubernetes.io/projected/e5b5b84d-ccf6-4683-af90-ba5aef6c821e-kube-api-access-855wx\") pod \"payload-processing-e2e-api-b-02a8fa-6987989665-drfl8\" (UID: \"e5b5b84d-ccf6-4683-af90-ba5aef6c821e\") " pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"
Oct 30 22:07:39.840165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:39.814763    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b5b84d-ccf6-4683-af90-ba5aef6c821e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-02a8fa-6987989665-drfl8\" (UID: \"e5b5b84d-ccf6-4683-af90-ba5aef6c821e\") " pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"
Oct 30 22:07:39.840787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:39.824231    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b5b84d-ccf6-4683-af90-ba5aef6c821e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-02a8fa-6987989665-drfl8\" (UID: \"e5b5b84d-ccf6-4683-af90-ba5aef6c821e\") " pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"
Oct 30 22:07:39.944162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:39.931092    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"]
Oct 30 22:07:39.979264 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:39.979171    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-855wx\" (UniqueName: \"kubernetes.io/projected/e5b5b84d-ccf6-4683-af90-ba5aef6c821e-kube-api-access-855wx\") pod \"payload-processing-e2e-api-b-02a8fa-6987989665-drfl8\" (UID: \"e5b5b84d-ccf6-4683-af90-ba5aef6c821e\") " pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"
Oct 30 22:07:40.133161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:40.132063    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"
Oct 30 22:07:40.744474 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:40.744442    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"]
Oct 30 22:07:40.745067 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:07:40.745039    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b5b84d_ccf6_4683_af90_ba5aef6c821e.slice/crio-891695c653c3cbfa42c935d365b94e4268d5d108e4cc49b174e50e39d233de67 WatchSource:0}: Error finding container 891695c653c3cbfa42c935d365b94e4268d5d108e4cc49b174e50e39d233de67: Status 404 returned error can't find the container with id 891695c653c3cbfa42c935d365b94e4268d5d108e4cc49b174e50e39d233de67
Oct 30 22:07:41.427453 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:41.427413    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8" event={"ID":"e5b5b84d-ccf6-4683-af90-ba5aef6c821e","Type":"ContainerStarted","Data":"7b4a88ba55a0c0ee2c46e6decae26e5fe821354cabb78d460b97e1bbdf835fad"}
Oct 30 22:07:41.427667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:41.427478    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8" event={"ID":"e5b5b84d-ccf6-4683-af90-ba5aef6c821e","Type":"ContainerStarted","Data":"891695c653c3cbfa42c935d365b94e4268d5d108e4cc49b174e50e39d233de67"}
Oct 30 22:07:41.427667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:41.427526    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"
Oct 30 22:07:41.549521 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:41.549460    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8" podStartSLOduration=2.5494451590000002 podStartE2EDuration="2.549445159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:41.546186784 +0000 UTC m=+1512.280549817" watchObservedRunningTime="2026-10-30 22:07:41.549445159 +0000 UTC m=+1512.283808190"
Oct 30 22:07:45.344752 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:45.344715    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"]
Oct 30 22:07:45.409162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:45.400088    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"]
Oct 30 22:07:45.409162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:45.405112    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"
Oct 30 22:07:45.493164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:45.490871    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80c6022-402a-4810-9396-39a9765237c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-479363-96bb58d89-lv8jl\" (UID: \"f80c6022-402a-4810-9396-39a9765237c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"
Oct 30 22:07:45.493164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:45.491066    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfjtj\" (UniqueName: \"kubernetes.io/projected/f80c6022-402a-4810-9396-39a9765237c9-kube-api-access-nfjtj\") pod \"payload-processing-e2e-api-b-479363-96bb58d89-lv8jl\" (UID: \"f80c6022-402a-4810-9396-39a9765237c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"
Oct 30 22:07:45.568170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:45.539656    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-479363\""
Oct 30 22:07:45.568554 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:45.568520    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-479363-dockercfg-fzppz\""
Oct 30 22:07:45.592652 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:45.592613    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80c6022-402a-4810-9396-39a9765237c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-479363-96bb58d89-lv8jl\" (UID: \"f80c6022-402a-4810-9396-39a9765237c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"
Oct 30 22:07:45.592894 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:45.592680    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfjtj\" (UniqueName: \"kubernetes.io/projected/f80c6022-402a-4810-9396-39a9765237c9-kube-api-access-nfjtj\") pod \"payload-processing-e2e-api-b-479363-96bb58d89-lv8jl\" (UID: \"f80c6022-402a-4810-9396-39a9765237c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"
Oct 30 22:07:45.593738 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:45.593709    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80c6022-402a-4810-9396-39a9765237c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-479363-96bb58d89-lv8jl\" (UID: \"f80c6022-402a-4810-9396-39a9765237c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"
Oct 30 22:07:45.650971 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:45.650896    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfjtj\" (UniqueName: \"kubernetes.io/projected/f80c6022-402a-4810-9396-39a9765237c9-kube-api-access-nfjtj\") pod \"payload-processing-e2e-api-b-479363-96bb58d89-lv8jl\" (UID: \"f80c6022-402a-4810-9396-39a9765237c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"
Oct 30 22:07:45.848348 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:45.848319    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"
Oct 30 22:07:46.227177 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:46.227145    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"]
Oct 30 22:07:46.227674 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:07:46.227650    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80c6022_402a_4810_9396_39a9765237c9.slice/crio-1e731010b7a695d415e9dfe0e2d7d198de79cc51aece5ac963171f07e90de5b8 WatchSource:0}: Error finding container 1e731010b7a695d415e9dfe0e2d7d198de79cc51aece5ac963171f07e90de5b8: Status 404 returned error can't find the container with id 1e731010b7a695d415e9dfe0e2d7d198de79cc51aece5ac963171f07e90de5b8
Oct 30 22:07:46.591605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:46.591570    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl" event={"ID":"f80c6022-402a-4810-9396-39a9765237c9","Type":"ContainerStarted","Data":"a2898b29173ac8095b6acbf5b4e992200d6436beec0ad1d6b995057d80fe3f85"}
Oct 30 22:07:46.592008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:46.591631    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl" event={"ID":"f80c6022-402a-4810-9396-39a9765237c9","Type":"ContainerStarted","Data":"1e731010b7a695d415e9dfe0e2d7d198de79cc51aece5ac963171f07e90de5b8"}
Oct 30 22:07:46.592008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:46.591702    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"
Oct 30 22:07:46.708291 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:46.708242    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl" podStartSLOduration=1.708222606 podStartE2EDuration="1.708222606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:46.706925915 +0000 UTC m=+1517.441288959" watchObservedRunningTime="2026-10-30 22:07:46.708222606 +0000 UTC m=+1517.442585637"
Oct 30 22:07:52.438002 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:52.437973    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"
Oct 30 22:07:57.601892 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:07:57.601862    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"
Oct 30 22:07:58.205096 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:07:58.205057    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:08:00.968754 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:00.968712    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"]
Oct 30 22:08:00.974788 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:00.974755    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:01.031634 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.031599    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"]
Oct 30 22:08:01.076056 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.076017    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d73ad7-serving-cert\""
Oct 30 22:08:01.138305 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.138268    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-openshift-service-ca\") pod \"maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:01.138487 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.138349    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7m7m\" (UniqueName: \"kubernetes.io/projected/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-kube-api-access-s7m7m\") pod \"maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:01.138643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.138621    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-metrics-certs\") pod \"maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:01.138708 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.138688    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-maas-api-tls\") pod \"maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:01.239929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.239832    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-metrics-certs\") pod \"maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:01.239929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.239887    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-maas-api-tls\") pod \"maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:01.240125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.240003    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-openshift-service-ca\") pod \"maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:01.240125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.240065    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7m7m\" (UniqueName: \"kubernetes.io/projected/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-kube-api-access-s7m7m\") pod \"maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:01.240847 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.240793    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-openshift-service-ca\") pod \"maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:01.242672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.242649    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-maas-api-tls\") pod \"maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:01.242814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.242791    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-metrics-certs\") pod \"maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:01.299429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.299401    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7m7m\" (UniqueName: \"kubernetes.io/projected/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-kube-api-access-s7m7m\") pod \"maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:01.590814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.590786    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:01.975546 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:01.975518    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"]
Oct 30 22:08:01.976087 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:08:01.976066    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5c81a1_c0f5_4c31_b1c5_ddd80a270e28.slice/crio-89f2dc124fdb273491d003a5e142d2093fb66305ace2df1f74ed0e9f7608d876 WatchSource:0}: Error finding container 89f2dc124fdb273491d003a5e142d2093fb66305ace2df1f74ed0e9f7608d876: Status 404 returned error can't find the container with id 89f2dc124fdb273491d003a5e142d2093fb66305ace2df1f74ed0e9f7608d876
Oct 30 22:08:02.712872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:02.712830    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m" event={"ID":"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28","Type":"ContainerStarted","Data":"89f2dc124fdb273491d003a5e142d2093fb66305ace2df1f74ed0e9f7608d876"}
Oct 30 22:08:03.721522 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:03.721477    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m" event={"ID":"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28","Type":"ContainerStarted","Data":"6bdb0884e5db2ba9f7be04b3c3b0bdd72785a7b25806270b7fad2cc986aca43c"}
Oct 30 22:08:03.721944 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:03.721584    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:03.839992 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:03.839928    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m" podStartSLOduration=2.8701478639999998 podStartE2EDuration="3.839909623s" totalImagesPullingTime="969.761759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:03.838329238 +0000 UTC m=+1534.572692269" watchObservedRunningTime="2026-10-30 22:08:03.839909623 +0000 UTC m=+1534.574272655"
Oct 30 22:08:09.735809 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:09.735781    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:08:28.208246 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:08:28.208210    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:08:50.113913 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:50.113852    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"]
Oct 30 22:08:50.152840 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:50.152808    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"
Oct 30 22:08:50.249621 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:50.248039    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-97ca39-dockercfg-6vq7r\""
Oct 30 22:08:50.249621 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:50.248125    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-97ca39\""
Oct 30 22:08:50.269396 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:50.264588    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee10de5-d1f3-47dd-a0e8-a97e11381b1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz\" (UID: \"7ee10de5-d1f3-47dd-a0e8-a97e11381b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"
Oct 30 22:08:50.269396 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:50.264731    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6fq4\" (UniqueName: \"kubernetes.io/projected/7ee10de5-d1f3-47dd-a0e8-a97e11381b1e-kube-api-access-k6fq4\") pod \"payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz\" (UID: \"7ee10de5-d1f3-47dd-a0e8-a97e11381b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"
Oct 30 22:08:50.269396 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:50.267810    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"]
Oct 30 22:08:50.391148 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:50.390070    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee10de5-d1f3-47dd-a0e8-a97e11381b1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz\" (UID: \"7ee10de5-d1f3-47dd-a0e8-a97e11381b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"
Oct 30 22:08:50.391508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:50.391483    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6fq4\" (UniqueName: \"kubernetes.io/projected/7ee10de5-d1f3-47dd-a0e8-a97e11381b1e-kube-api-access-k6fq4\") pod \"payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz\" (UID: \"7ee10de5-d1f3-47dd-a0e8-a97e11381b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"
Oct 30 22:08:50.393110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:50.393081    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee10de5-d1f3-47dd-a0e8-a97e11381b1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz\" (UID: \"7ee10de5-d1f3-47dd-a0e8-a97e11381b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"
Oct 30 22:08:50.527207 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:50.519794    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6fq4\" (UniqueName: \"kubernetes.io/projected/7ee10de5-d1f3-47dd-a0e8-a97e11381b1e-kube-api-access-k6fq4\") pod \"payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz\" (UID: \"7ee10de5-d1f3-47dd-a0e8-a97e11381b1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"
Oct 30 22:08:50.599162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:50.598446    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"
Oct 30 22:08:51.046545 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:51.046519    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"]
Oct 30 22:08:51.047655 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:08:51.047630    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee10de5_d1f3_47dd_a0e8_a97e11381b1e.slice/crio-c32f13292b6e883b3e5496c3aabf5cda741eebc884298d8d37fae2b4f9d0a5b8 WatchSource:0}: Error finding container c32f13292b6e883b3e5496c3aabf5cda741eebc884298d8d37fae2b4f9d0a5b8: Status 404 returned error can't find the container with id c32f13292b6e883b3e5496c3aabf5cda741eebc884298d8d37fae2b4f9d0a5b8
Oct 30 22:08:51.145099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:51.145069    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz" event={"ID":"7ee10de5-d1f3-47dd-a0e8-a97e11381b1e","Type":"ContainerStarted","Data":"ef8e2da9f6270901e7b9896b0ca7272f44564d7972e05e6e38ffaa97cefcf1b5"}
Oct 30 22:08:51.145570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:51.145119    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz" event={"ID":"7ee10de5-d1f3-47dd-a0e8-a97e11381b1e","Type":"ContainerStarted","Data":"c32f13292b6e883b3e5496c3aabf5cda741eebc884298d8d37fae2b4f9d0a5b8"}
Oct 30 22:08:51.145570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:51.145388    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"
Oct 30 22:08:51.283763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:08:51.283709    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz" podStartSLOduration=1.283694344 podStartE2EDuration="1.283694344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:51.279860331 +0000 UTC m=+1582.014223362" watchObservedRunningTime="2026-10-30 22:08:51.283694344 +0000 UTC m=+1582.018057375"
Oct 30 22:08:58.204998 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:08:58.204956    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:09:02.166745 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:02.166715    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"
Oct 30 22:09:11.472155 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.464381    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"]
Oct 30 22:09:11.502161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.495835    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.626168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.566760    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-podinfo\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.626168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.566898    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae0b720-525f-43fc-a94b-42e079ce85b4-istiod-ca-cert\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.626168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.566957    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-envoy\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.626168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.567004    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-token\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.626168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.567057    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-credential-socket\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.626168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.567172    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-workload-socket\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.626168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.567258    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-workload-certs\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.626168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.567290    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-data\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.626168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.567313    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2qpt\" (UniqueName: \"kubernetes.io/projected/cae0b720-525f-43fc-a94b-42e079ce85b4-kube-api-access-r2qpt\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.709156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.689278    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-podinfo\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.709156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.689411    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae0b720-525f-43fc-a94b-42e079ce85b4-istiod-ca-cert\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.709156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.689466    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-envoy\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.709156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.689508    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-token\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.709156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.689557    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-credential-socket\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.709156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.696226    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"]
Oct 30 22:09:11.709156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.696972    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae0b720-525f-43fc-a94b-42e079ce85b4-istiod-ca-cert\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.709156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.698344    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-workload-socket\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.709156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.698414    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-workload-certs\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.709156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.698448    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-data\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.709156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.698474    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qpt\" (UniqueName: \"kubernetes.io/projected/cae0b720-525f-43fc-a94b-42e079ce85b4-kube-api-access-r2qpt\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.709156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.700241    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-credential-socket\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.709156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.700480    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-workload-socket\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.709156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.700701    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-workload-certs\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.709156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.700914    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-data\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.783291 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.733417    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-podinfo\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.783291 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.739775    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-envoy\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.783291 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.767502    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2bc5e3-openshift-default-dockercfg-7gxqh\""
Oct 30 22:09:11.979859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.954718    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-token\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:11.985017 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:11.982947    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qpt\" (UniqueName: \"kubernetes.io/projected/cae0b720-525f-43fc-a94b-42e079ce85b4-kube-api-access-r2qpt\") pod \"e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") " pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:12.129107 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:12.129077    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:12.621471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:12.621448    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"]
Oct 30 22:09:12.623420 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:09:12.623388    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae0b720_525f_43fc_a94b_42e079ce85b4.slice/crio-9e4e54d969129e4c732d8f16e398824ebdcaba37770a6089d455a7400320fd98 WatchSource:0}: Error finding container 9e4e54d969129e4c732d8f16e398824ebdcaba37770a6089d455a7400320fd98: Status 404 returned error can't find the container with id 9e4e54d969129e4c732d8f16e398824ebdcaba37770a6089d455a7400320fd98
Oct 30 22:09:12.626928 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:12.626890    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:09:12.627020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:12.626980    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:09:12.627060 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:12.627021    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:09:13.429886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:13.429845    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq" event={"ID":"cae0b720-525f-43fc-a94b-42e079ce85b4","Type":"ContainerStarted","Data":"81d22899ad6575a630c03418d4ce4d82410837e40ed6c76cb1900ccfbd55e7ff"}
Oct 30 22:09:13.430085 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:13.429907    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq" event={"ID":"cae0b720-525f-43fc-a94b-42e079ce85b4","Type":"ContainerStarted","Data":"9e4e54d969129e4c732d8f16e398824ebdcaba37770a6089d455a7400320fd98"}
Oct 30 22:09:13.605552 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:13.605498    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq" podStartSLOduration=2.605480868 podStartE2EDuration="2.605480868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:13.60265172 +0000 UTC m=+1604.337014752" watchObservedRunningTime="2026-10-30 22:09:13.605480868 +0000 UTC m=+1604.339843988"
Oct 30 22:09:14.130049 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:14.130008    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:14.135980 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:14.135951    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:14.437554 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:14.437467    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:14.438648 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:14.438625    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:09:28.204354 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:09:28.204316    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:09:54.094805 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:54.094750    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"]
Oct 30 22:09:54.208765 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:54.208729    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"]
Oct 30 22:09:58.205593 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:09:58.205552    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:09:59.096948 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.096884    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v" podUID="5a70e922-07d2-475f-861d-a0a2f54c588f" containerName="payload-processing" containerID="cri-o://1337529090f27bf07d1a1eb0bf70aea3b511e14c4cb8cbdeb306a3eb1da708f1" gracePeriod=25
Oct 30 22:09:59.209566 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.209481    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6" podUID="44fd1f7f-a04d-4edc-8e59-0887fa39176a" containerName="payload-pre-processing" containerID="cri-o://002f2783f30653728e25bc3a39decfc3485c82702090142fa4c0ff08bf569cc7" gracePeriod=25
Oct 30 22:09:59.394880 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.394851    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"
Oct 30 22:09:59.456786 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.456756    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a70e922-07d2-475f-861d-a0a2f54c588f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a70e922-07d2-475f-861d-a0a2f54c588f-plugins-config-volume\") pod \"5a70e922-07d2-475f-861d-a0a2f54c588f\" (UID: \"5a70e922-07d2-475f-861d-a0a2f54c588f\") "
Oct 30 22:09:59.457028 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.456819    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a70e922-07d2-475f-861d-a0a2f54c588f-kube-api-access-wjhgk\" (UniqueName: \"kubernetes.io/projected/5a70e922-07d2-475f-861d-a0a2f54c588f-kube-api-access-wjhgk\") pod \"5a70e922-07d2-475f-861d-a0a2f54c588f\" (UID: \"5a70e922-07d2-475f-861d-a0a2f54c588f\") "
Oct 30 22:09:59.457262 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.457212    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a70e922-07d2-475f-861d-a0a2f54c588f-plugins-config-volume" pod "5a70e922-07d2-475f-861d-a0a2f54c588f" (UID: "5a70e922-07d2-475f-861d-a0a2f54c588f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:59.457711 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.457690    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a70e922-07d2-475f-861d-a0a2f54c588f-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:59.460480 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.460451    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a70e922-07d2-475f-861d-a0a2f54c588f-kube-api-access-wjhgk" pod "5a70e922-07d2-475f-861d-a0a2f54c588f" (UID: "5a70e922-07d2-475f-861d-a0a2f54c588f"). InnerVolumeSpecName "kube-api-access-wjhgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:59.519849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.519823    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"
Oct 30 22:09:59.558578 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.558540    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44fd1f7f-a04d-4edc-8e59-0887fa39176a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fd1f7f-a04d-4edc-8e59-0887fa39176a-plugins-config-volume\") pod \"44fd1f7f-a04d-4edc-8e59-0887fa39176a\" (UID: \"44fd1f7f-a04d-4edc-8e59-0887fa39176a\") "
Oct 30 22:09:59.558796 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.558615    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44fd1f7f-a04d-4edc-8e59-0887fa39176a-kube-api-access-8q2ds\" (UniqueName: \"kubernetes.io/projected/44fd1f7f-a04d-4edc-8e59-0887fa39176a-kube-api-access-8q2ds\") pod \"44fd1f7f-a04d-4edc-8e59-0887fa39176a\" (UID: \"44fd1f7f-a04d-4edc-8e59-0887fa39176a\") "
Oct 30 22:09:59.559055 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.559025    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44fd1f7f-a04d-4edc-8e59-0887fa39176a-plugins-config-volume" pod "44fd1f7f-a04d-4edc-8e59-0887fa39176a" (UID: "44fd1f7f-a04d-4edc-8e59-0887fa39176a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:59.559385 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.559361    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fd1f7f-a04d-4edc-8e59-0887fa39176a-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:59.559489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.559391    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjhgk\" (UniqueName: \"kubernetes.io/projected/5a70e922-07d2-475f-861d-a0a2f54c588f-kube-api-access-wjhgk\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:59.561463 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.561434    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fd1f7f-a04d-4edc-8e59-0887fa39176a-kube-api-access-8q2ds" pod "44fd1f7f-a04d-4edc-8e59-0887fa39176a" (UID: "44fd1f7f-a04d-4edc-8e59-0887fa39176a"). InnerVolumeSpecName "kube-api-access-8q2ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:59.660544 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.660467    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q2ds\" (UniqueName: \"kubernetes.io/projected/44fd1f7f-a04d-4edc-8e59-0887fa39176a-kube-api-access-8q2ds\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:59.838520 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.838488    4553 generic.go:356] "Generic (PLEG): container finished" podID="44fd1f7f-a04d-4edc-8e59-0887fa39176a" containerID="002f2783f30653728e25bc3a39decfc3485c82702090142fa4c0ff08bf569cc7" exitCode=0
Oct 30 22:09:59.838700 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.838535    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"
Oct 30 22:09:59.838700 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.838574    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6" event={"ID":"44fd1f7f-a04d-4edc-8e59-0887fa39176a","Type":"ContainerDied","Data":"002f2783f30653728e25bc3a39decfc3485c82702090142fa4c0ff08bf569cc7"}
Oct 30 22:09:59.838700 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.838628    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6" event={"ID":"44fd1f7f-a04d-4edc-8e59-0887fa39176a","Type":"ContainerDied","Data":"0fc95febf77d670589246f9913d82832cd173f1423a3adc3d160b9d5fe63a585"}
Oct 30 22:09:59.838700 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.838651    4553 scope.go:122] "RemoveContainer" containerID="002f2783f30653728e25bc3a39decfc3485c82702090142fa4c0ff08bf569cc7"
Oct 30 22:09:59.840602 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.840553    4553 generic.go:356] "Generic (PLEG): container finished" podID="5a70e922-07d2-475f-861d-a0a2f54c588f" containerID="1337529090f27bf07d1a1eb0bf70aea3b511e14c4cb8cbdeb306a3eb1da708f1" exitCode=0
Oct 30 22:09:59.840602 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.840580    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v" event={"ID":"5a70e922-07d2-475f-861d-a0a2f54c588f","Type":"ContainerDied","Data":"1337529090f27bf07d1a1eb0bf70aea3b511e14c4cb8cbdeb306a3eb1da708f1"}
Oct 30 22:09:59.840834 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.840625    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"
Oct 30 22:09:59.840999 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.840630    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v" event={"ID":"5a70e922-07d2-475f-861d-a0a2f54c588f","Type":"ContainerDied","Data":"5198c4ac7ddbb3ce1f3d7e33e8a1ee867c90243161d38675eb7d430957ecdf4c"}
Oct 30 22:09:59.855263 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.855094    4553 scope.go:122] "RemoveContainer" containerID="002f2783f30653728e25bc3a39decfc3485c82702090142fa4c0ff08bf569cc7"
Oct 30 22:09:59.855677 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:09:59.855649    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002f2783f30653728e25bc3a39decfc3485c82702090142fa4c0ff08bf569cc7\": container with ID starting with 002f2783f30653728e25bc3a39decfc3485c82702090142fa4c0ff08bf569cc7 not found: ID does not exist" containerID="002f2783f30653728e25bc3a39decfc3485c82702090142fa4c0ff08bf569cc7"
Oct 30 22:09:59.855790 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.855689    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002f2783f30653728e25bc3a39decfc3485c82702090142fa4c0ff08bf569cc7"} err="failed to get container status \"002f2783f30653728e25bc3a39decfc3485c82702090142fa4c0ff08bf569cc7\": rpc error: code = NotFound desc = could not find container \"002f2783f30653728e25bc3a39decfc3485c82702090142fa4c0ff08bf569cc7\": container with ID starting with 002f2783f30653728e25bc3a39decfc3485c82702090142fa4c0ff08bf569cc7 not found: ID does not exist"
Oct 30 22:09:59.855790 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.855707    4553 scope.go:122] "RemoveContainer" containerID="1337529090f27bf07d1a1eb0bf70aea3b511e14c4cb8cbdeb306a3eb1da708f1"
Oct 30 22:09:59.865659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.865635    4553 scope.go:122] "RemoveContainer" containerID="1337529090f27bf07d1a1eb0bf70aea3b511e14c4cb8cbdeb306a3eb1da708f1"
Oct 30 22:09:59.865919 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:09:59.865903    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1337529090f27bf07d1a1eb0bf70aea3b511e14c4cb8cbdeb306a3eb1da708f1\": container with ID starting with 1337529090f27bf07d1a1eb0bf70aea3b511e14c4cb8cbdeb306a3eb1da708f1 not found: ID does not exist" containerID="1337529090f27bf07d1a1eb0bf70aea3b511e14c4cb8cbdeb306a3eb1da708f1"
Oct 30 22:09:59.865972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:09:59.865927    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1337529090f27bf07d1a1eb0bf70aea3b511e14c4cb8cbdeb306a3eb1da708f1"} err="failed to get container status \"1337529090f27bf07d1a1eb0bf70aea3b511e14c4cb8cbdeb306a3eb1da708f1\": rpc error: code = NotFound desc = could not find container \"1337529090f27bf07d1a1eb0bf70aea3b511e14c4cb8cbdeb306a3eb1da708f1\": container with ID starting with 1337529090f27bf07d1a1eb0bf70aea3b511e14c4cb8cbdeb306a3eb1da708f1 not found: ID does not exist"
Oct 30 22:10:00.011945 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:00.011860    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"]
Oct 30 22:10:00.021020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:00.020992    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ac4a7-698788cc4f-p2v7v"]
Oct 30 22:10:00.142962 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:00.142927    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"]
Oct 30 22:10:00.161098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:00.161062    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ac4a7-99d55f65f-dttp6"]
Oct 30 22:10:01.394228 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:01.394195    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44fd1f7f-a04d-4edc-8e59-0887fa39176a" path="/var/lib/kubelet/pods/44fd1f7f-a04d-4edc-8e59-0887fa39176a/volumes"
Oct 30 22:10:01.394665 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:01.394650    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a70e922-07d2-475f-861d-a0a2f54c588f" path="/var/lib/kubelet/pods/5a70e922-07d2-475f-861d-a0a2f54c588f/volumes"
Oct 30 22:10:05.073534 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:05.073500    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"]
Oct 30 22:10:05.073970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:05.073735    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" podUID="eedadc43-2c7f-465a-a834-f1871f020e61" containerName="istio-proxy" containerID="cri-o://a77e9f6fce43fbde3e46a85d21f6d36039152f2cd870607296c609d18d778f91" gracePeriod=30
Oct 30 22:10:05.124241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:05.124200    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"]
Oct 30 22:10:05.124569 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:05.124523    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp" podUID="49818a62-7ddf-43b3-9aa8-cf0aadf73516" containerName="istio-proxy" containerID="cri-o://7799182f07e44817b9391a7954c253083b73455deafa5d1bf674396bd1b86854" gracePeriod=30
Oct 30 22:10:05.128512 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:05.128484    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"]
Oct 30 22:10:05.128773 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:05.128725    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7" podUID="44783759-0591-4a2c-b850-97dff6f9085b" containerName="istio-proxy" containerID="cri-o://5481510b4d60c0f17ea910fd479c00adabb5430793c19cd89f62a9837cf32082" gracePeriod=30
Oct 30 22:10:05.174489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:05.174438    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"]
Oct 30 22:10:05.174696 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:05.174649    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q" podUID="5c189afe-afe8-4736-8e80-e321da9d7268" containerName="istio-proxy" containerID="cri-o://3dff47f256c389f1992996fbfa0a9310e9ee0dd7d475744aedd6411c3fa71151" gracePeriod=30
Oct 30 22:10:10.280319 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.280285    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"]
Oct 30 22:10:10.280784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.280475    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr" podUID="3267e456-b8e8-43be-9360-f75afeaadb7b" containerName="istio-proxy" containerID="cri-o://a51d900eb15052a2ff2ebd314670c8d17ef846d2ddc232c189a5b7d835935f1c" gracePeriod=30
Oct 30 22:10:10.440532 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.440106    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:10:10.557184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.550707    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-credential-socket\") pod \"eedadc43-2c7f-465a-a834-f1871f020e61\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") "
Oct 30 22:10:10.557184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.550766    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eedadc43-2c7f-465a-a834-f1871f020e61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedadc43-2c7f-465a-a834-f1871f020e61-istiod-ca-cert\") pod \"eedadc43-2c7f-465a-a834-f1871f020e61\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") "
Oct 30 22:10:10.557184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.550905    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eedadc43-2c7f-465a-a834-f1871f020e61-kube-api-access-cdkr8\" (UniqueName: \"kubernetes.io/projected/eedadc43-2c7f-465a-a834-f1871f020e61-kube-api-access-cdkr8\") pod \"eedadc43-2c7f-465a-a834-f1871f020e61\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") "
Oct 30 22:10:10.557184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.550979    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-workload-socket\") pod \"eedadc43-2c7f-465a-a834-f1871f020e61\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") "
Oct 30 22:10:10.557184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.551051    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-istio-data\") pod \"eedadc43-2c7f-465a-a834-f1871f020e61\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") "
Oct 30 22:10:10.557184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.551149    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-istio-envoy\") pod \"eedadc43-2c7f-465a-a834-f1871f020e61\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") "
Oct 30 22:10:10.557184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.551188    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eedadc43-2c7f-465a-a834-f1871f020e61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedadc43-2c7f-465a-a834-f1871f020e61-istio-podinfo\") pod \"eedadc43-2c7f-465a-a834-f1871f020e61\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") "
Oct 30 22:10:10.557184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.551262    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eedadc43-2c7f-465a-a834-f1871f020e61-istio-token\" (UniqueName: \"kubernetes.io/projected/eedadc43-2c7f-465a-a834-f1871f020e61-istio-token\") pod \"eedadc43-2c7f-465a-a834-f1871f020e61\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") "
Oct 30 22:10:10.557184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.551342    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-workload-certs\") pod \"eedadc43-2c7f-465a-a834-f1871f020e61\" (UID: \"eedadc43-2c7f-465a-a834-f1871f020e61\") "
Oct 30 22:10:10.557184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.553509    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:10:10.557184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.553851    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-istio-data" pod "eedadc43-2c7f-465a-a834-f1871f020e61" (UID: "eedadc43-2c7f-465a-a834-f1871f020e61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.557184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.554887    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-credential-socket" pod "eedadc43-2c7f-465a-a834-f1871f020e61" (UID: "eedadc43-2c7f-465a-a834-f1871f020e61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.557184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.556986    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eedadc43-2c7f-465a-a834-f1871f020e61-istiod-ca-cert" pod "eedadc43-2c7f-465a-a834-f1871f020e61" (UID: "eedadc43-2c7f-465a-a834-f1871f020e61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:10.558004 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.557239    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-workload-certs" pod "eedadc43-2c7f-465a-a834-f1871f020e61" (UID: "eedadc43-2c7f-465a-a834-f1871f020e61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.558004 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.557306    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-workload-socket" pod "eedadc43-2c7f-465a-a834-f1871f020e61" (UID: "eedadc43-2c7f-465a-a834-f1871f020e61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.558808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.558680    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eedadc43-2c7f-465a-a834-f1871f020e61-istio-podinfo" pod "eedadc43-2c7f-465a-a834-f1871f020e61" (UID: "eedadc43-2c7f-465a-a834-f1871f020e61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:10.558867 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.558803    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eedadc43-2c7f-465a-a834-f1871f020e61-kube-api-access-cdkr8" pod "eedadc43-2c7f-465a-a834-f1871f020e61" (UID: "eedadc43-2c7f-465a-a834-f1871f020e61"). InnerVolumeSpecName "kube-api-access-cdkr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:10.561872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.561634    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eedadc43-2c7f-465a-a834-f1871f020e61-istio-token" pod "eedadc43-2c7f-465a-a834-f1871f020e61" (UID: "eedadc43-2c7f-465a-a834-f1871f020e61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:10.582832 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.582711    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:10:10.584548 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.584514    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-istio-envoy" pod "eedadc43-2c7f-465a-a834-f1871f020e61" (UID: "eedadc43-2c7f-465a-a834-f1871f020e61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.586750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.586724    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 22:10:10.652833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.652739    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44783759-0591-4a2c-b850-97dff6f9085b-istio-token\" (UniqueName: \"kubernetes.io/projected/44783759-0591-4a2c-b850-97dff6f9085b-istio-token\") pod \"44783759-0591-4a2c-b850-97dff6f9085b\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") "
Oct 30 22:10:10.652833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.652788    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-workload-certs\") pod \"44783759-0591-4a2c-b850-97dff6f9085b\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") "
Oct 30 22:10:10.653153 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.652846    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-credential-socket\") pod \"44783759-0591-4a2c-b850-97dff6f9085b\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") "
Oct 30 22:10:10.653153 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.652882    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44783759-0591-4a2c-b850-97dff6f9085b-kube-api-access-jgz6w\" (UniqueName: \"kubernetes.io/projected/44783759-0591-4a2c-b850-97dff6f9085b-kube-api-access-jgz6w\") pod \"44783759-0591-4a2c-b850-97dff6f9085b\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") "
Oct 30 22:10:10.653153 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.652962    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-workload-socket\") pod \"44783759-0591-4a2c-b850-97dff6f9085b\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") "
Oct 30 22:10:10.653153 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.653109    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-istio-data\") pod \"44783759-0591-4a2c-b850-97dff6f9085b\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") "
Oct 30 22:10:10.653388 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.653241    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-credential-socket" pod "44783759-0591-4a2c-b850-97dff6f9085b" (UID: "44783759-0591-4a2c-b850-97dff6f9085b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.653474 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.653453    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-workload-certs" pod "44783759-0591-4a2c-b850-97dff6f9085b" (UID: "44783759-0591-4a2c-b850-97dff6f9085b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.653527 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.653475    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-workload-socket" pod "44783759-0591-4a2c-b850-97dff6f9085b" (UID: "44783759-0591-4a2c-b850-97dff6f9085b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.653691 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.653670    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-istio-data" pod "44783759-0591-4a2c-b850-97dff6f9085b" (UID: "44783759-0591-4a2c-b850-97dff6f9085b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.653902 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.653887    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44783759-0591-4a2c-b850-97dff6f9085b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44783759-0591-4a2c-b850-97dff6f9085b-istiod-ca-cert\") pod \"44783759-0591-4a2c-b850-97dff6f9085b\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") "
Oct 30 22:10:10.654002 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.653962    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-istio-envoy\") pod \"44783759-0591-4a2c-b850-97dff6f9085b\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") "
Oct 30 22:10:10.654065 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.654053    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44783759-0591-4a2c-b850-97dff6f9085b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44783759-0591-4a2c-b850-97dff6f9085b-istio-podinfo\") pod \"44783759-0591-4a2c-b850-97dff6f9085b\" (UID: \"44783759-0591-4a2c-b850-97dff6f9085b\") "
Oct 30 22:10:10.654419 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.654367    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44783759-0591-4a2c-b850-97dff6f9085b-istiod-ca-cert" pod "44783759-0591-4a2c-b850-97dff6f9085b" (UID: "44783759-0591-4a2c-b850-97dff6f9085b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:10.654994 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.654971    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdkr8\" (UniqueName: \"kubernetes.io/projected/eedadc43-2c7f-465a-a834-f1871f020e61-kube-api-access-cdkr8\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.655106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.655000    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.655106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.655018    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.655106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.655032    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.655106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.655046    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedadc43-2c7f-465a-a834-f1871f020e61-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.655106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.655060    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedadc43-2c7f-465a-a834-f1871f020e61-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.655106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.655074    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.655106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.655088    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.655106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.655101    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.655587 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.655116    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedadc43-2c7f-465a-a834-f1871f020e61-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.655587 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.655147    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.655587 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.655160    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedadc43-2c7f-465a-a834-f1871f020e61-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.655587 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.655172    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.655587 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.655191    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44783759-0591-4a2c-b850-97dff6f9085b-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.656005 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.655970    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44783759-0591-4a2c-b850-97dff6f9085b-istio-token" pod "44783759-0591-4a2c-b850-97dff6f9085b" (UID: "44783759-0591-4a2c-b850-97dff6f9085b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:10.656067 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.656003    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44783759-0591-4a2c-b850-97dff6f9085b-kube-api-access-jgz6w" pod "44783759-0591-4a2c-b850-97dff6f9085b" (UID: "44783759-0591-4a2c-b850-97dff6f9085b"). InnerVolumeSpecName "kube-api-access-jgz6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:10.656900 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.656881    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-istio-envoy" pod "44783759-0591-4a2c-b850-97dff6f9085b" (UID: "44783759-0591-4a2c-b850-97dff6f9085b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.657327 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.657305    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44783759-0591-4a2c-b850-97dff6f9085b-istio-podinfo" pod "44783759-0591-4a2c-b850-97dff6f9085b" (UID: "44783759-0591-4a2c-b850-97dff6f9085b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:10.755849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.755811    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-data\") pod \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") "
Oct 30 22:10:10.756062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.755862    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-workload-socket\") pod \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") "
Oct 30 22:10:10.756062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.755936    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-workload-certs\") pod \"5c189afe-afe8-4736-8e80-e321da9d7268\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") "
Oct 30 22:10:10.756062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.755974    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-credential-socket\") pod \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") "
Oct 30 22:10:10.756062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756005    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c189afe-afe8-4736-8e80-e321da9d7268-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c189afe-afe8-4736-8e80-e321da9d7268-istio-podinfo\") pod \"5c189afe-afe8-4736-8e80-e321da9d7268\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") "
Oct 30 22:10:10.756062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756056    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c189afe-afe8-4736-8e80-e321da9d7268-kube-api-access-mm4zl\" (UniqueName: \"kubernetes.io/projected/5c189afe-afe8-4736-8e80-e321da9d7268-kube-api-access-mm4zl\") pod \"5c189afe-afe8-4736-8e80-e321da9d7268\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") "
Oct 30 22:10:10.756355 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756054    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-data" pod "49818a62-7ddf-43b3-9aa8-cf0aadf73516" (UID: "49818a62-7ddf-43b3-9aa8-cf0aadf73516"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.756355 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756101    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-envoy\") pod \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") "
Oct 30 22:10:10.756355 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756178    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-workload-socket" pod "49818a62-7ddf-43b3-9aa8-cf0aadf73516" (UID: "49818a62-7ddf-43b3-9aa8-cf0aadf73516"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.756355 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756197    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-workload-certs" pod "5c189afe-afe8-4736-8e80-e321da9d7268" (UID: "5c189afe-afe8-4736-8e80-e321da9d7268"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.756355 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756308    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-istio-envoy\") pod \"5c189afe-afe8-4736-8e80-e321da9d7268\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") "
Oct 30 22:10:10.756355 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756343    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-credential-socket" pod "49818a62-7ddf-43b3-9aa8-cf0aadf73516" (UID: "49818a62-7ddf-43b3-9aa8-cf0aadf73516"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.756625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756365    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-credential-socket\") pod \"5c189afe-afe8-4736-8e80-e321da9d7268\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") "
Oct 30 22:10:10.756625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756429    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-istio-data\") pod \"5c189afe-afe8-4736-8e80-e321da9d7268\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") "
Oct 30 22:10:10.756625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756537    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c189afe-afe8-4736-8e80-e321da9d7268-istio-token\" (UniqueName: \"kubernetes.io/projected/5c189afe-afe8-4736-8e80-e321da9d7268-istio-token\") pod \"5c189afe-afe8-4736-8e80-e321da9d7268\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") "
Oct 30 22:10:10.756625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756577    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49818a62-7ddf-43b3-9aa8-cf0aadf73516-kube-api-access-rzgjd\" (UniqueName: \"kubernetes.io/projected/49818a62-7ddf-43b3-9aa8-cf0aadf73516-kube-api-access-rzgjd\") pod \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") "
Oct 30 22:10:10.756784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756656    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-credential-socket" pod "5c189afe-afe8-4736-8e80-e321da9d7268" (UID: "5c189afe-afe8-4736-8e80-e321da9d7268"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.756784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756663    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istiod-ca-cert\") pod \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") "
Oct 30 22:10:10.756784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756714    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c189afe-afe8-4736-8e80-e321da9d7268-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c189afe-afe8-4736-8e80-e321da9d7268-istiod-ca-cert\") pod \"5c189afe-afe8-4736-8e80-e321da9d7268\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") "
Oct 30 22:10:10.756784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756724    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-istio-data" pod "5c189afe-afe8-4736-8e80-e321da9d7268" (UID: "5c189afe-afe8-4736-8e80-e321da9d7268"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.756784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.756759    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-workload-certs\") pod \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") "
Oct 30 22:10:10.757074 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.757056    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-workload-socket\") pod \"5c189afe-afe8-4736-8e80-e321da9d7268\" (UID: \"5c189afe-afe8-4736-8e80-e321da9d7268\") "
Oct 30 22:10:10.757116 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.757101    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-podinfo\") pod \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") "
Oct 30 22:10:10.757231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.757197    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-token\" (UniqueName: \"kubernetes.io/projected/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-token\") pod \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\" (UID: \"49818a62-7ddf-43b3-9aa8-cf0aadf73516\") "
Oct 30 22:10:10.757491 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.757464    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-workload-socket" pod "5c189afe-afe8-4736-8e80-e321da9d7268" (UID: "5c189afe-afe8-4736-8e80-e321da9d7268"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.757491 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.757385    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c189afe-afe8-4736-8e80-e321da9d7268-istiod-ca-cert" pod "5c189afe-afe8-4736-8e80-e321da9d7268" (UID: "5c189afe-afe8-4736-8e80-e321da9d7268"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:10.757780 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.757663    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-workload-certs" pod "49818a62-7ddf-43b3-9aa8-cf0aadf73516" (UID: "49818a62-7ddf-43b3-9aa8-cf0aadf73516"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.757996 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.757858    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istiod-ca-cert" pod "49818a62-7ddf-43b3-9aa8-cf0aadf73516" (UID: "49818a62-7ddf-43b3-9aa8-cf0aadf73516"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:10.758084 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.758063    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.758182 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.758084    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgz6w\" (UniqueName: \"kubernetes.io/projected/44783759-0591-4a2c-b850-97dff6f9085b-kube-api-access-jgz6w\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.758182 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.758099    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.758182 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.758113    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.758182 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.758128    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44783759-0591-4a2c-b850-97dff6f9085b-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.758409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.758190    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.758409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.758204    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44783759-0591-4a2c-b850-97dff6f9085b-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.758409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.758219    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.758409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.758234    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c189afe-afe8-4736-8e80-e321da9d7268-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.758409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.758248    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.758409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.758261    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.758409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.758274    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44783759-0591-4a2c-b850-97dff6f9085b-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.758409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.758288    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.758409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.758304    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.759347 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.759319    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-envoy" pod "49818a62-7ddf-43b3-9aa8-cf0aadf73516" (UID: "49818a62-7ddf-43b3-9aa8-cf0aadf73516"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.759485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.759342    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c189afe-afe8-4736-8e80-e321da9d7268-kube-api-access-mm4zl" pod "5c189afe-afe8-4736-8e80-e321da9d7268" (UID: "5c189afe-afe8-4736-8e80-e321da9d7268"). InnerVolumeSpecName "kube-api-access-mm4zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:10.759485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.759443    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-istio-envoy" pod "5c189afe-afe8-4736-8e80-e321da9d7268" (UID: "5c189afe-afe8-4736-8e80-e321da9d7268"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:10.759732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.759715    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49818a62-7ddf-43b3-9aa8-cf0aadf73516-kube-api-access-rzgjd" pod "49818a62-7ddf-43b3-9aa8-cf0aadf73516" (UID: "49818a62-7ddf-43b3-9aa8-cf0aadf73516"). InnerVolumeSpecName "kube-api-access-rzgjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:10.760215 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.760108    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-token" pod "49818a62-7ddf-43b3-9aa8-cf0aadf73516" (UID: "49818a62-7ddf-43b3-9aa8-cf0aadf73516"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:10.760215 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.760177    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c189afe-afe8-4736-8e80-e321da9d7268-istio-podinfo" pod "5c189afe-afe8-4736-8e80-e321da9d7268" (UID: "5c189afe-afe8-4736-8e80-e321da9d7268"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:10.760486 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.760463    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-podinfo" pod "49818a62-7ddf-43b3-9aa8-cf0aadf73516" (UID: "49818a62-7ddf-43b3-9aa8-cf0aadf73516"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:10.761711 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.761691    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c189afe-afe8-4736-8e80-e321da9d7268-istio-token" pod "5c189afe-afe8-4736-8e80-e321da9d7268" (UID: "5c189afe-afe8-4736-8e80-e321da9d7268"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:10.860084 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.860049    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c189afe-afe8-4736-8e80-e321da9d7268-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.860295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.860088    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzgjd\" (UniqueName: \"kubernetes.io/projected/49818a62-7ddf-43b3-9aa8-cf0aadf73516-kube-api-access-rzgjd\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.860295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.860107    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.860295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.860121    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.860295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.860156    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c189afe-afe8-4736-8e80-e321da9d7268-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.860295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.860172    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm4zl\" (UniqueName: \"kubernetes.io/projected/5c189afe-afe8-4736-8e80-e321da9d7268-kube-api-access-mm4zl\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.860295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.860185    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49818a62-7ddf-43b3-9aa8-cf0aadf73516-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.860295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.860199    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c189afe-afe8-4736-8e80-e321da9d7268-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:10.939311 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.939214    4553 generic.go:356] "Generic (PLEG): container finished" podID="eedadc43-2c7f-465a-a834-f1871f020e61" containerID="a77e9f6fce43fbde3e46a85d21f6d36039152f2cd870607296c609d18d778f91" exitCode=0
Oct 30 22:10:10.939311 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.939268    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"
Oct 30 22:10:10.939545 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.939302    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" event={"ID":"eedadc43-2c7f-465a-a834-f1871f020e61","Type":"ContainerDied","Data":"a77e9f6fce43fbde3e46a85d21f6d36039152f2cd870607296c609d18d778f91"}
Oct 30 22:10:10.939545 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.939365    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g" event={"ID":"eedadc43-2c7f-465a-a834-f1871f020e61","Type":"ContainerDied","Data":"9ecfe3ffc8733aa218425faed2708cff2acb61f6bad521adf94d165a5ade9d29"}
Oct 30 22:10:10.939545 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.939400    4553 scope.go:122] "RemoveContainer" containerID="a77e9f6fce43fbde3e46a85d21f6d36039152f2cd870607296c609d18d778f91"
Oct 30 22:10:10.943056 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.943029    4553 generic.go:356] "Generic (PLEG): container finished" podID="5c189afe-afe8-4736-8e80-e321da9d7268" containerID="3dff47f256c389f1992996fbfa0a9310e9ee0dd7d475744aedd6411c3fa71151" exitCode=0
Oct 30 22:10:10.943209 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.943153    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q" event={"ID":"5c189afe-afe8-4736-8e80-e321da9d7268","Type":"ContainerDied","Data":"3dff47f256c389f1992996fbfa0a9310e9ee0dd7d475744aedd6411c3fa71151"}
Oct 30 22:10:10.943287 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.943249    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q" event={"ID":"5c189afe-afe8-4736-8e80-e321da9d7268","Type":"ContainerDied","Data":"696b5c97f32851f6edff9beadc4b250f6ebdb59b22ca4f9efe693ad6023593e5"}
Oct 30 22:10:10.943374 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.943358    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"
Oct 30 22:10:10.950992 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.950966    4553 generic.go:356] "Generic (PLEG): container finished" podID="49818a62-7ddf-43b3-9aa8-cf0aadf73516" containerID="7799182f07e44817b9391a7954c253083b73455deafa5d1bf674396bd1b86854" exitCode=0
Oct 30 22:10:10.951192 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.951087    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp" event={"ID":"49818a62-7ddf-43b3-9aa8-cf0aadf73516","Type":"ContainerDied","Data":"7799182f07e44817b9391a7954c253083b73455deafa5d1bf674396bd1b86854"}
Oct 30 22:10:10.951311 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.951200    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp" event={"ID":"49818a62-7ddf-43b3-9aa8-cf0aadf73516","Type":"ContainerDied","Data":"89e595557c3e04f3ad9f7fddc4a71ca7f56c6782384d9bb842ec977cfc309862"}
Oct 30 22:10:10.951452 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.951313    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"
Oct 30 22:10:10.956395 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.956378    4553 generic.go:356] "Generic (PLEG): container finished" podID="44783759-0591-4a2c-b850-97dff6f9085b" containerID="5481510b4d60c0f17ea910fd479c00adabb5430793c19cd89f62a9837cf32082" exitCode=0
Oct 30 22:10:10.956473 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.956439    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"
Oct 30 22:10:10.956680 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.956655    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7" event={"ID":"44783759-0591-4a2c-b850-97dff6f9085b","Type":"ContainerDied","Data":"5481510b4d60c0f17ea910fd479c00adabb5430793c19cd89f62a9837cf32082"}
Oct 30 22:10:10.956770 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.956711    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7" event={"ID":"44783759-0591-4a2c-b850-97dff6f9085b","Type":"ContainerDied","Data":"67f1304bc7d653404e43e227370b9a14395eab48a2f0e3d414ee19e6b16ccf3f"}
Oct 30 22:10:10.956832 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.956802    4553 scope.go:122] "RemoveContainer" containerID="a77e9f6fce43fbde3e46a85d21f6d36039152f2cd870607296c609d18d778f91"
Oct 30 22:10:10.960320 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:10:10.960293    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77e9f6fce43fbde3e46a85d21f6d36039152f2cd870607296c609d18d778f91\": container with ID starting with a77e9f6fce43fbde3e46a85d21f6d36039152f2cd870607296c609d18d778f91 not found: ID does not exist" containerID="a77e9f6fce43fbde3e46a85d21f6d36039152f2cd870607296c609d18d778f91"
Oct 30 22:10:10.960430 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.960331    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77e9f6fce43fbde3e46a85d21f6d36039152f2cd870607296c609d18d778f91"} err="failed to get container status \"a77e9f6fce43fbde3e46a85d21f6d36039152f2cd870607296c609d18d778f91\": rpc error: code = NotFound desc = could not find container \"a77e9f6fce43fbde3e46a85d21f6d36039152f2cd870607296c609d18d778f91\": container with ID starting with a77e9f6fce43fbde3e46a85d21f6d36039152f2cd870607296c609d18d778f91 not found: ID does not exist"
Oct 30 22:10:10.960430 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.960356    4553 scope.go:122] "RemoveContainer" containerID="3dff47f256c389f1992996fbfa0a9310e9ee0dd7d475744aedd6411c3fa71151"
Oct 30 22:10:10.974737 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.974716    4553 scope.go:122] "RemoveContainer" containerID="5db73e8a9d255a13bd7b722d14e7a376efd66a26fdf799304296c1b2f88040d0"
Oct 30 22:10:10.988201 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.988187    4553 scope.go:122] "RemoveContainer" containerID="3dff47f256c389f1992996fbfa0a9310e9ee0dd7d475744aedd6411c3fa71151"
Oct 30 22:10:10.988607 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:10:10.988583    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dff47f256c389f1992996fbfa0a9310e9ee0dd7d475744aedd6411c3fa71151\": container with ID starting with 3dff47f256c389f1992996fbfa0a9310e9ee0dd7d475744aedd6411c3fa71151 not found: ID does not exist" containerID="3dff47f256c389f1992996fbfa0a9310e9ee0dd7d475744aedd6411c3fa71151"
Oct 30 22:10:10.988733 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.988614    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dff47f256c389f1992996fbfa0a9310e9ee0dd7d475744aedd6411c3fa71151"} err="failed to get container status \"3dff47f256c389f1992996fbfa0a9310e9ee0dd7d475744aedd6411c3fa71151\": rpc error: code = NotFound desc = could not find container \"3dff47f256c389f1992996fbfa0a9310e9ee0dd7d475744aedd6411c3fa71151\": container with ID starting with 3dff47f256c389f1992996fbfa0a9310e9ee0dd7d475744aedd6411c3fa71151 not found: ID does not exist"
Oct 30 22:10:10.988733 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.988632    4553 scope.go:122] "RemoveContainer" containerID="5db73e8a9d255a13bd7b722d14e7a376efd66a26fdf799304296c1b2f88040d0"
Oct 30 22:10:10.989021 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:10:10.988927    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5db73e8a9d255a13bd7b722d14e7a376efd66a26fdf799304296c1b2f88040d0\": container with ID starting with 5db73e8a9d255a13bd7b722d14e7a376efd66a26fdf799304296c1b2f88040d0 not found: ID does not exist" containerID="5db73e8a9d255a13bd7b722d14e7a376efd66a26fdf799304296c1b2f88040d0"
Oct 30 22:10:10.989126 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.989032    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5db73e8a9d255a13bd7b722d14e7a376efd66a26fdf799304296c1b2f88040d0"} err="failed to get container status \"5db73e8a9d255a13bd7b722d14e7a376efd66a26fdf799304296c1b2f88040d0\": rpc error: code = NotFound desc = could not find container \"5db73e8a9d255a13bd7b722d14e7a376efd66a26fdf799304296c1b2f88040d0\": container with ID starting with 5db73e8a9d255a13bd7b722d14e7a376efd66a26fdf799304296c1b2f88040d0 not found: ID does not exist"
Oct 30 22:10:10.989126 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.989049    4553 scope.go:122] "RemoveContainer" containerID="7799182f07e44817b9391a7954c253083b73455deafa5d1bf674396bd1b86854"
Oct 30 22:10:10.999565 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.999534    4553 scope.go:122] "RemoveContainer" containerID="7799182f07e44817b9391a7954c253083b73455deafa5d1bf674396bd1b86854"
Oct 30 22:10:10.999923 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:10:10.999899    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7799182f07e44817b9391a7954c253083b73455deafa5d1bf674396bd1b86854\": container with ID starting with 7799182f07e44817b9391a7954c253083b73455deafa5d1bf674396bd1b86854 not found: ID does not exist" containerID="7799182f07e44817b9391a7954c253083b73455deafa5d1bf674396bd1b86854"
Oct 30 22:10:11.000038 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.999936    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7799182f07e44817b9391a7954c253083b73455deafa5d1bf674396bd1b86854"} err="failed to get container status \"7799182f07e44817b9391a7954c253083b73455deafa5d1bf674396bd1b86854\": rpc error: code = NotFound desc = could not find container \"7799182f07e44817b9391a7954c253083b73455deafa5d1bf674396bd1b86854\": container with ID starting with 7799182f07e44817b9391a7954c253083b73455deafa5d1bf674396bd1b86854 not found: ID does not exist"
Oct 30 22:10:11.000038 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:10.999964    4553 scope.go:122] "RemoveContainer" containerID="5481510b4d60c0f17ea910fd479c00adabb5430793c19cd89f62a9837cf32082"
Oct 30 22:10:11.010350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:11.010331    4553 scope.go:122] "RemoveContainer" containerID="5481510b4d60c0f17ea910fd479c00adabb5430793c19cd89f62a9837cf32082"
Oct 30 22:10:11.010683 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:10:11.010661    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5481510b4d60c0f17ea910fd479c00adabb5430793c19cd89f62a9837cf32082\": container with ID starting with 5481510b4d60c0f17ea910fd479c00adabb5430793c19cd89f62a9837cf32082 not found: ID does not exist" containerID="5481510b4d60c0f17ea910fd479c00adabb5430793c19cd89f62a9837cf32082"
Oct 30 22:10:11.010782 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:11.010690    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5481510b4d60c0f17ea910fd479c00adabb5430793c19cd89f62a9837cf32082"} err="failed to get container status \"5481510b4d60c0f17ea910fd479c00adabb5430793c19cd89f62a9837cf32082\": rpc error: code = NotFound desc = could not find container \"5481510b4d60c0f17ea910fd479c00adabb5430793c19cd89f62a9837cf32082\": container with ID starting with 5481510b4d60c0f17ea910fd479c00adabb5430793c19cd89f62a9837cf32082 not found: ID does not exist"
Oct 30 22:10:11.127341 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:11.123424    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"]
Oct 30 22:10:11.133150 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:11.133094    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-dh2fp"]
Oct 30 22:10:11.244804 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:11.244715    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"]
Oct 30 22:10:11.251405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:11.251373    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-lxdn7"]
Oct 30 22:10:11.368267 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:11.368227    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"]
Oct 30 22:10:11.371862 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:11.371836    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-pmb6g"]
Oct 30 22:10:11.394799 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:11.394768    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44783759-0591-4a2c-b850-97dff6f9085b" path="/var/lib/kubelet/pods/44783759-0591-4a2c-b850-97dff6f9085b/volumes"
Oct 30 22:10:11.395341 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:11.395327    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49818a62-7ddf-43b3-9aa8-cf0aadf73516" path="/var/lib/kubelet/pods/49818a62-7ddf-43b3-9aa8-cf0aadf73516/volumes"
Oct 30 22:10:11.395772 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:11.395760    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eedadc43-2c7f-465a-a834-f1871f020e61" path="/var/lib/kubelet/pods/eedadc43-2c7f-465a-a834-f1871f020e61/volumes"
Oct 30 22:10:11.483932 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:11.483895    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"]
Oct 30 22:10:11.488007 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:11.487976    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d2a3e0-openshift-default-c6495b579-cxz2q"]
Oct 30 22:10:12.038540 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.038497    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"]
Oct 30 22:10:12.480003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.479952    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"]
Oct 30 22:10:12.481713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.481679    4553 cpu_manager.go:414] "containerMap: removing container" podUID="49818a62-7ddf-43b3-9aa8-cf0aadf73516" containerName="istio-proxy"
Oct 30 22:10:12.481907 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.481890    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49818a62-7ddf-43b3-9aa8-cf0aadf73516" containerName="istio-proxy"
Oct 30 22:10:12.482090 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.482047    4553 cpu_manager.go:414] "containerMap: removing container" podUID="5a70e922-07d2-475f-861d-a0a2f54c588f" containerName="payload-processing"
Oct 30 22:10:12.482292 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.482276    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a70e922-07d2-475f-861d-a0a2f54c588f" containerName="payload-processing"
Oct 30 22:10:12.482471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.482457    4553 cpu_manager.go:414] "containerMap: removing container" podUID="eedadc43-2c7f-465a-a834-f1871f020e61" containerName="istio-proxy"
Oct 30 22:10:12.482579 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.482566    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eedadc43-2c7f-465a-a834-f1871f020e61" containerName="istio-proxy"
Oct 30 22:10:12.482668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.482656    4553 cpu_manager.go:414] "containerMap: removing container" podUID="44783759-0591-4a2c-b850-97dff6f9085b" containerName="istio-proxy"
Oct 30 22:10:12.482752 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.482739    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44783759-0591-4a2c-b850-97dff6f9085b" containerName="istio-proxy"
Oct 30 22:10:12.482830 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.482820    4553 cpu_manager.go:414] "containerMap: removing container" podUID="5c189afe-afe8-4736-8e80-e321da9d7268" containerName="istio-proxy"
Oct 30 22:10:12.482920 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.482909    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c189afe-afe8-4736-8e80-e321da9d7268" containerName="istio-proxy"
Oct 30 22:10:12.483011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.483000    4553 cpu_manager.go:414] "containerMap: removing container" podUID="5c189afe-afe8-4736-8e80-e321da9d7268" containerName="istio-proxy"
Oct 30 22:10:12.483087 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.483077    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c189afe-afe8-4736-8e80-e321da9d7268" containerName="istio-proxy"
Oct 30 22:10:12.483197 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.483186    4553 cpu_manager.go:414] "containerMap: removing container" podUID="44fd1f7f-a04d-4edc-8e59-0887fa39176a" containerName="payload-pre-processing"
Oct 30 22:10:12.483764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.483746    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44fd1f7f-a04d-4edc-8e59-0887fa39176a" containerName="payload-pre-processing"
Oct 30 22:10:12.484156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.484099    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a70e922-07d2-475f-861d-a0a2f54c588f" containerName="payload-processing"
Oct 30 22:10:12.484287 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.484274    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="eedadc43-2c7f-465a-a834-f1871f020e61" containerName="istio-proxy"
Oct 30 22:10:12.484369 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.484358    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="44783759-0591-4a2c-b850-97dff6f9085b" containerName="istio-proxy"
Oct 30 22:10:12.484472 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.484462    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c189afe-afe8-4736-8e80-e321da9d7268" containerName="istio-proxy"
Oct 30 22:10:12.484560 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.484550    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c189afe-afe8-4736-8e80-e321da9d7268" containerName="istio-proxy"
Oct 30 22:10:12.484635 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.484626    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="44fd1f7f-a04d-4edc-8e59-0887fa39176a" containerName="payload-pre-processing"
Oct 30 22:10:12.484717 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.484708    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="49818a62-7ddf-43b3-9aa8-cf0aadf73516" containerName="istio-proxy"
Oct 30 22:10:12.504493 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.504466    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:12.560629 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.559618    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2bc5e3-serving-cert\""
Oct 30 22:10:12.631160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.630993    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-maas-api-tls\") pod \"maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:12.631160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.631054    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-metrics-certs\") pod \"maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:12.631420 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.631278    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72b872a-65b7-4923-b5f9-fbb01be82e41-openshift-service-ca\") pod \"maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:12.652160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.631456    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlc2\" (UniqueName: \"kubernetes.io/projected/f72b872a-65b7-4923-b5f9-fbb01be82e41-kube-api-access-qmlc2\") pod \"maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:12.652160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.645029    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"]
Oct 30 22:10:12.767252 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.762251    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72b872a-65b7-4923-b5f9-fbb01be82e41-openshift-service-ca\") pod \"maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:12.767252 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.762313    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlc2\" (UniqueName: \"kubernetes.io/projected/f72b872a-65b7-4923-b5f9-fbb01be82e41-kube-api-access-qmlc2\") pod \"maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:12.767252 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.762564    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-maas-api-tls\") pod \"maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:12.767252 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.762625    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-metrics-certs\") pod \"maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:12.767252 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:10:12.763917    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-serving-cert: secret "maas-api-e2e-api-b-2bc5e3-serving-cert" not found
Oct 30 22:10:12.767252 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:10:12.763993    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-maas-api-tls podName:f72b872a-65b7-4923-b5f9-fbb01be82e41 nodeName:}" failed. No retries permitted until 2026-10-30 22:10:13.263969895 +0000 UTC m=+1663.998332930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-maas-api-tls") pod "maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv" (UID: "f72b872a-65b7-4923-b5f9-fbb01be82e41") : secret "maas-api-e2e-api-b-2bc5e3-serving-cert" not found
Oct 30 22:10:12.767252 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.765618    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72b872a-65b7-4923-b5f9-fbb01be82e41-openshift-service-ca\") pod \"maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:12.814177 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.786287    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-metrics-certs\") pod \"maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:12.977161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:12.910103    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlc2\" (UniqueName: \"kubernetes.io/projected/f72b872a-65b7-4923-b5f9-fbb01be82e41-kube-api-access-qmlc2\") pod \"maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:13.321386 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:13.321340    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-maas-api-tls\") pod \"maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:13.321615 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:10:13.321516    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-serving-cert: secret "maas-api-e2e-api-b-2bc5e3-serving-cert" not found
Oct 30 22:10:13.321615 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:10:13.321597    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-maas-api-tls podName:f72b872a-65b7-4923-b5f9-fbb01be82e41 nodeName:}" failed. No retries permitted until 2026-10-30 22:10:14.321577992 +0000 UTC m=+1665.055941015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-maas-api-tls") pod "maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv" (UID: "f72b872a-65b7-4923-b5f9-fbb01be82e41") : secret "maas-api-e2e-api-b-2bc5e3-serving-cert" not found
Oct 30 22:10:13.394802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:13.394766    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c189afe-afe8-4736-8e80-e321da9d7268" path="/var/lib/kubelet/pods/5c189afe-afe8-4736-8e80-e321da9d7268/volumes"
Oct 30 22:10:14.372447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:14.372413    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-maas-api-tls\") pod \"maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:14.374995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:14.374960    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-maas-api-tls\") pod \"maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:14.429708 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:14.429666    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:14.735198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:14.731282    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:10:14.735198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:14.731359    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr" podUID="3267e456-b8e8-43be-9360-f75afeaadb7b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:10:14.923575 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:14.923550    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"]
Oct 30 22:10:14.969077 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:14.969028    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:10:15.051693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.051649    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv" event={"ID":"f72b872a-65b7-4923-b5f9-fbb01be82e41","Type":"ContainerStarted","Data":"9dff6f7d176e5a3bf7992c2c04e86f18b985292f4d4921ae197137c860a09879"}
Oct 30 22:10:15.563970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.563898    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:10:15.699735 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.699706    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3267e456-b8e8-43be-9360-f75afeaadb7b-kube-api-access-dcmgk\" (UniqueName: \"kubernetes.io/projected/3267e456-b8e8-43be-9360-f75afeaadb7b-kube-api-access-dcmgk\") pod \"3267e456-b8e8-43be-9360-f75afeaadb7b\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") "
Oct 30 22:10:15.699915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.699770    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-podinfo\") pod \"3267e456-b8e8-43be-9360-f75afeaadb7b\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") "
Oct 30 22:10:15.699915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.699803    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-workload-socket\") pod \"3267e456-b8e8-43be-9360-f75afeaadb7b\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") "
Oct 30 22:10:15.700060 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.699932    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3267e456-b8e8-43be-9360-f75afeaadb7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3267e456-b8e8-43be-9360-f75afeaadb7b-istiod-ca-cert\") pod \"3267e456-b8e8-43be-9360-f75afeaadb7b\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") "
Oct 30 22:10:15.700060 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.699949    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-envoy\") pod \"3267e456-b8e8-43be-9360-f75afeaadb7b\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") "
Oct 30 22:10:15.700060 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.699985    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-workload-certs\") pod \"3267e456-b8e8-43be-9360-f75afeaadb7b\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") "
Oct 30 22:10:15.700324 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.700078    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-data\") pod \"3267e456-b8e8-43be-9360-f75afeaadb7b\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") "
Oct 30 22:10:15.700324 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.700233    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-workload-socket" pod "3267e456-b8e8-43be-9360-f75afeaadb7b" (UID: "3267e456-b8e8-43be-9360-f75afeaadb7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.700436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.700347    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3267e456-b8e8-43be-9360-f75afeaadb7b-istiod-ca-cert" pod "3267e456-b8e8-43be-9360-f75afeaadb7b" (UID: "3267e456-b8e8-43be-9360-f75afeaadb7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:15.700436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.700384    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-data" pod "3267e456-b8e8-43be-9360-f75afeaadb7b" (UID: "3267e456-b8e8-43be-9360-f75afeaadb7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.700436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.700393    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-workload-certs" pod "3267e456-b8e8-43be-9360-f75afeaadb7b" (UID: "3267e456-b8e8-43be-9360-f75afeaadb7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.700588 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.700481    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-credential-socket\") pod \"3267e456-b8e8-43be-9360-f75afeaadb7b\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") "
Oct 30 22:10:15.700588 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.700509    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-token\" (UniqueName: \"kubernetes.io/projected/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-token\") pod \"3267e456-b8e8-43be-9360-f75afeaadb7b\" (UID: \"3267e456-b8e8-43be-9360-f75afeaadb7b\") "
Oct 30 22:10:15.700798 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.700749    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-credential-socket" pod "3267e456-b8e8-43be-9360-f75afeaadb7b" (UID: "3267e456-b8e8-43be-9360-f75afeaadb7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.701298 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.701275    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.701298 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.701300    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.701659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.701309    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.701659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.701319    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3267e456-b8e8-43be-9360-f75afeaadb7b-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.701659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.701331    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.702477 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.702454    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-podinfo" pod "3267e456-b8e8-43be-9360-f75afeaadb7b" (UID: "3267e456-b8e8-43be-9360-f75afeaadb7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:15.702776 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.702750    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-envoy" pod "3267e456-b8e8-43be-9360-f75afeaadb7b" (UID: "3267e456-b8e8-43be-9360-f75afeaadb7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:15.702776 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.702769    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3267e456-b8e8-43be-9360-f75afeaadb7b-kube-api-access-dcmgk" pod "3267e456-b8e8-43be-9360-f75afeaadb7b" (UID: "3267e456-b8e8-43be-9360-f75afeaadb7b"). InnerVolumeSpecName "kube-api-access-dcmgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:15.703065 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.703049    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-token" pod "3267e456-b8e8-43be-9360-f75afeaadb7b" (UID: "3267e456-b8e8-43be-9360-f75afeaadb7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:15.803815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.803777    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcmgk\" (UniqueName: \"kubernetes.io/projected/3267e456-b8e8-43be-9360-f75afeaadb7b-kube-api-access-dcmgk\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.803815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.803818    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.804014 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.803836    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.804014 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:15.803850    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3267e456-b8e8-43be-9360-f75afeaadb7b-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:16.164743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:16.164638    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv" event={"ID":"f72b872a-65b7-4923-b5f9-fbb01be82e41","Type":"ContainerStarted","Data":"aee38380a5d5d5227fa5d405651d0b1d73f9aef6732031aafc4cfd8f6c7dfb30"}
Oct 30 22:10:16.173192 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:16.168120    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:16.173192 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:16.170153    4553 generic.go:356] "Generic (PLEG): container finished" podID="3267e456-b8e8-43be-9360-f75afeaadb7b" containerID="a51d900eb15052a2ff2ebd314670c8d17ef846d2ddc232c189a5b7d835935f1c" exitCode=0
Oct 30 22:10:16.173192 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:16.170240    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr" event={"ID":"3267e456-b8e8-43be-9360-f75afeaadb7b","Type":"ContainerDied","Data":"a51d900eb15052a2ff2ebd314670c8d17ef846d2ddc232c189a5b7d835935f1c"}
Oct 30 22:10:16.173192 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:16.170288    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr" event={"ID":"3267e456-b8e8-43be-9360-f75afeaadb7b","Type":"ContainerDied","Data":"80c1dcbfb0cd2cec58022693ba0282444e140e23c0c37b68c36b86b5f9c1094a"}
Oct 30 22:10:16.173192 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:16.170321    4553 scope.go:122] "RemoveContainer" containerID="a51d900eb15052a2ff2ebd314670c8d17ef846d2ddc232c189a5b7d835935f1c"
Oct 30 22:10:16.173192 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:16.170485    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"
Oct 30 22:10:16.200545 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:16.200522    4553 scope.go:122] "RemoveContainer" containerID="a51d900eb15052a2ff2ebd314670c8d17ef846d2ddc232c189a5b7d835935f1c"
Oct 30 22:10:16.200907 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:10:16.200881    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a51d900eb15052a2ff2ebd314670c8d17ef846d2ddc232c189a5b7d835935f1c\": container with ID starting with a51d900eb15052a2ff2ebd314670c8d17ef846d2ddc232c189a5b7d835935f1c not found: ID does not exist" containerID="a51d900eb15052a2ff2ebd314670c8d17ef846d2ddc232c189a5b7d835935f1c"
Oct 30 22:10:16.200983 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:16.200918    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51d900eb15052a2ff2ebd314670c8d17ef846d2ddc232c189a5b7d835935f1c"} err="failed to get container status \"a51d900eb15052a2ff2ebd314670c8d17ef846d2ddc232c189a5b7d835935f1c\": rpc error: code = NotFound desc = could not find container \"a51d900eb15052a2ff2ebd314670c8d17ef846d2ddc232c189a5b7d835935f1c\": container with ID starting with a51d900eb15052a2ff2ebd314670c8d17ef846d2ddc232c189a5b7d835935f1c not found: ID does not exist"
Oct 30 22:10:16.298692 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:16.298620    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv" podStartSLOduration=3.4086347200000002 podStartE2EDuration="4.298596667s" totalImagesPullingTime="889.961947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:16.292917143 +0000 UTC m=+1667.027280175" watchObservedRunningTime="2026-10-30 22:10:16.298596667 +0000 UTC m=+1667.032959699"
Oct 30 22:10:16.472523 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:16.472388    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"]
Oct 30 22:10:16.476507 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:16.476475    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-7xnrr"]
Oct 30 22:10:17.039480 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:17.039409    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l" podUID="767d4fe8-a9f3-4627-bd29-68732f291535" containerName="payload-processing" containerID="cri-o://11e14ef318a24a3b6782c9c0351393d0428204c64684d72bb55d974da1a44eab" gracePeriod=25
Oct 30 22:10:17.180503 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:17.180470    4553 generic.go:356] "Generic (PLEG): container finished" podID="767d4fe8-a9f3-4627-bd29-68732f291535" containerID="11e14ef318a24a3b6782c9c0351393d0428204c64684d72bb55d974da1a44eab" exitCode=0
Oct 30 22:10:17.180696 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:17.180564    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l" event={"ID":"767d4fe8-a9f3-4627-bd29-68732f291535","Type":"ContainerDied","Data":"11e14ef318a24a3b6782c9c0351393d0428204c64684d72bb55d974da1a44eab"}
Oct 30 22:10:17.365746 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:17.365720    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"
Oct 30 22:10:17.383558 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:17.383521    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767d4fe8-a9f3-4627-bd29-68732f291535-kube-api-access-f8csh\" (UniqueName: \"kubernetes.io/projected/767d4fe8-a9f3-4627-bd29-68732f291535-kube-api-access-f8csh\") pod \"767d4fe8-a9f3-4627-bd29-68732f291535\" (UID: \"767d4fe8-a9f3-4627-bd29-68732f291535\") "
Oct 30 22:10:17.383558 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:17.383571    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767d4fe8-a9f3-4627-bd29-68732f291535-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767d4fe8-a9f3-4627-bd29-68732f291535-plugins-config-volume\") pod \"767d4fe8-a9f3-4627-bd29-68732f291535\" (UID: \"767d4fe8-a9f3-4627-bd29-68732f291535\") "
Oct 30 22:10:17.384032 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:17.384000    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767d4fe8-a9f3-4627-bd29-68732f291535-plugins-config-volume" pod "767d4fe8-a9f3-4627-bd29-68732f291535" (UID: "767d4fe8-a9f3-4627-bd29-68732f291535"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:17.384408 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:17.384387    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767d4fe8-a9f3-4627-bd29-68732f291535-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:17.385962 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:17.385942    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767d4fe8-a9f3-4627-bd29-68732f291535-kube-api-access-f8csh" pod "767d4fe8-a9f3-4627-bd29-68732f291535" (UID: "767d4fe8-a9f3-4627-bd29-68732f291535"). InnerVolumeSpecName "kube-api-access-f8csh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:17.395356 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:17.395323    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3267e456-b8e8-43be-9360-f75afeaadb7b" path="/var/lib/kubelet/pods/3267e456-b8e8-43be-9360-f75afeaadb7b/volumes"
Oct 30 22:10:17.486914 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:17.486172    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8csh\" (UniqueName: \"kubernetes.io/projected/767d4fe8-a9f3-4627-bd29-68732f291535-kube-api-access-f8csh\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:18.189472 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:18.189436    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"
Oct 30 22:10:18.189472 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:18.189460    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l" event={"ID":"767d4fe8-a9f3-4627-bd29-68732f291535","Type":"ContainerDied","Data":"418bc8cd1b180d43cb75656868ffd512cb36c36f9c4f6e9d78304460c63b944c"}
Oct 30 22:10:18.190012 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:18.189639    4553 scope.go:122] "RemoveContainer" containerID="11e14ef318a24a3b6782c9c0351393d0428204c64684d72bb55d974da1a44eab"
Oct 30 22:10:18.357682 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:18.357649    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"]
Oct 30 22:10:18.361097 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:18.361076    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-da0f3d-cc6c6bdc9-p5r7l"]
Oct 30 22:10:19.394975 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:19.394943    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="767d4fe8-a9f3-4627-bd29-68732f291535" path="/var/lib/kubelet/pods/767d4fe8-a9f3-4627-bd29-68732f291535/volumes"
Oct 30 22:10:23.195452 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:23.195420    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:10:28.203477 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:10:28.203439    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:10:43.089713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:43.089675    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"]
Oct 30 22:10:43.090310 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:43.089885    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh" podUID="9aa7dcc5-d42a-4f74-8452-542b6ed5b404" containerName="istio-proxy" containerID="cri-o://cac70d97ff7a51af1415f4f99c60ac2b942416a6ab116aae2ab65b45a24425f9" gracePeriod=30
Oct 30 22:10:48.110124 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.110093    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 22:10:48.110513 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.110175    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh" podUID="9aa7dcc5-d42a-4f74-8452-542b6ed5b404" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 22:10:48.437946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.437849    4553 generic.go:356] "Generic (PLEG): container finished" podID="9aa7dcc5-d42a-4f74-8452-542b6ed5b404" containerID="cac70d97ff7a51af1415f4f99c60ac2b942416a6ab116aae2ab65b45a24425f9" exitCode=0
Oct 30 22:10:48.437946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.437923    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh" event={"ID":"9aa7dcc5-d42a-4f74-8452-542b6ed5b404","Type":"ContainerDied","Data":"cac70d97ff7a51af1415f4f99c60ac2b942416a6ab116aae2ab65b45a24425f9"}
Oct 30 22:10:48.479551 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.479521    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:10:48.678232 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.678196    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-credential-socket\") pod \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") "
Oct 30 22:10:48.678411 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.678336    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-kube-api-access-wmgzh\" (UniqueName: \"kubernetes.io/projected/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-kube-api-access-wmgzh\") pod \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") "
Oct 30 22:10:48.678469 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.678431    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-workload-socket\") pod \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") "
Oct 30 22:10:48.678520 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.678472    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-token\" (UniqueName: \"kubernetes.io/projected/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-token\") pod \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") "
Oct 30 22:10:48.678520 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.678488    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-credential-socket" pod "9aa7dcc5-d42a-4f74-8452-542b6ed5b404" (UID: "9aa7dcc5-d42a-4f74-8452-542b6ed5b404"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.678520 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.678511    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istiod-ca-cert\") pod \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") "
Oct 30 22:10:48.678689 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.678589    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-data\") pod \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") "
Oct 30 22:10:48.678689 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.678682    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-podinfo\") pod \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") "
Oct 30 22:10:48.678795 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.678723    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-workload-certs\") pod \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") "
Oct 30 22:10:48.678795 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.678754    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-workload-socket" pod "9aa7dcc5-d42a-4f74-8452-542b6ed5b404" (UID: "9aa7dcc5-d42a-4f74-8452-542b6ed5b404"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.678795 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.678767    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-envoy\") pod \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\" (UID: \"9aa7dcc5-d42a-4f74-8452-542b6ed5b404\") "
Oct 30 22:10:48.679234 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.679004    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-data" pod "9aa7dcc5-d42a-4f74-8452-542b6ed5b404" (UID: "9aa7dcc5-d42a-4f74-8452-542b6ed5b404"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.679401 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.679380    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istiod-ca-cert" pod "9aa7dcc5-d42a-4f74-8452-542b6ed5b404" (UID: "9aa7dcc5-d42a-4f74-8452-542b6ed5b404"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:48.679634 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.679610    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-workload-certs" pod "9aa7dcc5-d42a-4f74-8452-542b6ed5b404" (UID: "9aa7dcc5-d42a-4f74-8452-542b6ed5b404"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.679749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.679724    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.679805 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.679757    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.679805 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.679772    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.679805 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.679786    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.681473 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.681444    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-token" pod "9aa7dcc5-d42a-4f74-8452-542b6ed5b404" (UID: "9aa7dcc5-d42a-4f74-8452-542b6ed5b404"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:48.682526 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.682494    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-podinfo" pod "9aa7dcc5-d42a-4f74-8452-542b6ed5b404" (UID: "9aa7dcc5-d42a-4f74-8452-542b6ed5b404"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:48.682695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.682553    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-envoy" pod "9aa7dcc5-d42a-4f74-8452-542b6ed5b404" (UID: "9aa7dcc5-d42a-4f74-8452-542b6ed5b404"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:48.682695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.682628    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-kube-api-access-wmgzh" pod "9aa7dcc5-d42a-4f74-8452-542b6ed5b404" (UID: "9aa7dcc5-d42a-4f74-8452-542b6ed5b404"). InnerVolumeSpecName "kube-api-access-wmgzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:48.781668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.781628    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.781668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.781665    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.781935 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.781678    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.781935 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.781690    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmgzh\" (UniqueName: \"kubernetes.io/projected/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-kube-api-access-wmgzh\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:48.781935 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:48.781704    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9aa7dcc5-d42a-4f74-8452-542b6ed5b404-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:49.445678 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:49.445648    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"
Oct 30 22:10:49.446093 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:49.445653    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh" event={"ID":"9aa7dcc5-d42a-4f74-8452-542b6ed5b404","Type":"ContainerDied","Data":"337e8a54728561090b9d30e404f62b15b2bb8cdcbf0e2d7d16f30ffab05b1447"}
Oct 30 22:10:49.446093 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:49.445780    4553 scope.go:122] "RemoveContainer" containerID="cac70d97ff7a51af1415f4f99c60ac2b942416a6ab116aae2ab65b45a24425f9"
Oct 30 22:10:49.667075 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:49.666988    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"]
Oct 30 22:10:49.717947 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:49.717908    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-ptlsh"]
Oct 30 22:10:51.395994 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:51.395958    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aa7dcc5-d42a-4f74-8452-542b6ed5b404" path="/var/lib/kubelet/pods/9aa7dcc5-d42a-4f74-8452-542b6ed5b404/volumes"
Oct 30 22:10:55.541359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:55.541319    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"]
Oct 30 22:10:55.541791 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:55.541542    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b" podUID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" containerName="istio-proxy" containerID="cri-o://859ba0b5db24ba75490f7df78c49586274caf8f859d10d5c8d027b31d0bb8619" gracePeriod=30
Oct 30 22:10:55.591827 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:55.591787    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"]
Oct 30 22:10:55.592083 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:55.592057    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp" podUID="fa482183-5d9f-461a-af5d-b30fd2ed8428" containerName="istio-proxy" containerID="cri-o://e9ba3d7bc6a24adbc52da5c485bd54a4d072889f45276618a26e740b76244f85" gracePeriod=30
Oct 30 22:10:55.595305 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:55.595268    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"]
Oct 30 22:10:55.595482 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:55.595459    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n" podUID="4bb761f4-75aa-481c-ad0a-0cfc782028ac" containerName="istio-proxy" containerID="cri-o://509b5b63acc19168ffc8d8fd98093ce9f9c2462a18c2e495b4b143667c7ee5b7" gracePeriod=30
Oct 30 22:10:55.642450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:55.642419    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"]
Oct 30 22:10:55.642630 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:55.642615    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr" podUID="3e4c8521-f6ec-4d79-8946-045048bdad39" containerName="istio-proxy" containerID="cri-o://39f6245dd87a5e20de41fd6e8f0012d20d66d2b8f8159277760d2c423c4923d1" gracePeriod=30
Oct 30 22:10:58.207354 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:10:58.207313    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:10:58.273293 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:58.273220    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"]
Oct 30 22:10:59.756109 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:59.755224    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"]
Oct 30 22:10:59.756109 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:59.755515    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy" containerID="cri-o://a49f50aa9f216766b38da6c11287d01bfb2b90a7750f943277d496726956cae4" gracePeriod=30
Oct 30 22:10:59.815160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:59.806498    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"]
Oct 30 22:10:59.815160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:59.806777    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy" containerID="cri-o://8e8a20b3f6e0915714ac7b35c7bfe1792148ae612743285943093523989f819a" gracePeriod=30
Oct 30 22:10:59.944606 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:59.944099    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"]
Oct 30 22:10:59.944606 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:59.944423    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj" podUID="20775493-c504-4316-9d6f-816952c83e9a" containerName="istio-proxy" containerID="cri-o://1ba57b2b607d4a4ec2a6c0a409d6c24898fcc8a932d16d197dd41e8999182789" gracePeriod=30
Oct 30 22:10:59.946666 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:59.945059    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"]
Oct 30 22:10:59.946666 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:10:59.945305    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z" podUID="c2b6e606-c225-4dd2-94c0-e8bfe42ee109" containerName="istio-proxy" containerID="cri-o://8a10b957163558bea4ea4fb9711fcbd115b449a585cfc38f9b5ad1fd658e2d27" gracePeriod=30
Oct 30 22:11:00.745072 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.745030    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 30 22:11:00.745258 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.745092    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n" podUID="4bb761f4-75aa-481c-ad0a-0cfc782028ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 30 22:11:00.800180 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.785671    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 22:11:00.800180 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.785815    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp" podUID="fa482183-5d9f-461a-af5d-b30fd2ed8428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 22:11:00.894769 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.894732    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 22:11:00.962918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.962876    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-data\") pod \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") "
Oct 30 22:11:00.963114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.962983    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-token\" (UniqueName: \"kubernetes.io/projected/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-token\") pod \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") "
Oct 30 22:11:00.963114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.963057    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-credential-socket\") pod \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") "
Oct 30 22:11:00.963317 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.963196    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-kube-api-access-tzpw6\" (UniqueName: \"kubernetes.io/projected/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-kube-api-access-tzpw6\") pod \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") "
Oct 30 22:11:00.963317 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.963297    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-data" pod "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" (UID: "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:00.963424 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.963307    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-workload-socket\") pod \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") "
Oct 30 22:11:00.963497 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.963458    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-podinfo\") pod \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") "
Oct 30 22:11:00.963563 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.963506    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-credential-socket" pod "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" (UID: "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:00.963563 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.963533    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-envoy\") pod \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") "
Oct 30 22:11:00.963709 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.963688    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-workload-certs\") pod \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") "
Oct 30 22:11:00.963786 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.963768    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istiod-ca-cert\") pod \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\" (UID: \"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3\") "
Oct 30 22:11:00.964849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.964802    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-workload-socket" pod "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" (UID: "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:00.964957 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.964925    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:00.965029 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.964969    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:00.965086 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.965032    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-workload-certs" pod "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" (UID: "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:00.965165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.965156    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istiod-ca-cert" pod "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" (UID: "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:00.977427 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.977371    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-kube-api-access-tzpw6" pod "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" (UID: "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3"). InnerVolumeSpecName "kube-api-access-tzpw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:00.977595 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.977565    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-envoy" pod "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" (UID: "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:00.977721 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.977701    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-token" pod "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" (UID: "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:00.980334 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:00.980303    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-podinfo" pod "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" (UID: "2747809c-ab61-4e18-8cd3-e6bcfe0de5c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:01.017171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.014221    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:11:01.063623 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.063590    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:11:01.065791 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.065762    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-workload-certs\") pod \"fa482183-5d9f-461a-af5d-b30fd2ed8428\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") "
Oct 30 22:11:01.065926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.065868    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa482183-5d9f-461a-af5d-b30fd2ed8428-kube-api-access-b6zj6\" (UniqueName: \"kubernetes.io/projected/fa482183-5d9f-461a-af5d-b30fd2ed8428-kube-api-access-b6zj6\") pod \"fa482183-5d9f-461a-af5d-b30fd2ed8428\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") "
Oct 30 22:11:01.065926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.065910    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-workload-socket\") pod \"fa482183-5d9f-461a-af5d-b30fd2ed8428\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") "
Oct 30 22:11:01.066045 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.065945    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-data\") pod \"fa482183-5d9f-461a-af5d-b30fd2ed8428\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") "
Oct 30 22:11:01.066045 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.065995    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-podinfo\") pod \"fa482183-5d9f-461a-af5d-b30fd2ed8428\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") "
Oct 30 22:11:01.066198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.066057    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-token\" (UniqueName: \"kubernetes.io/projected/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-token\") pod \"fa482183-5d9f-461a-af5d-b30fd2ed8428\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") "
Oct 30 22:11:01.066198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.066098    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-workload-certs" pod "fa482183-5d9f-461a-af5d-b30fd2ed8428" (UID: "fa482183-5d9f-461a-af5d-b30fd2ed8428"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:01.066198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.066154    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-envoy\") pod \"fa482183-5d9f-461a-af5d-b30fd2ed8428\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") "
Oct 30 22:11:01.066361 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.066288    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-credential-socket\") pod \"fa482183-5d9f-461a-af5d-b30fd2ed8428\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") "
Oct 30 22:11:01.066400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.066385    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa482183-5d9f-461a-af5d-b30fd2ed8428-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa482183-5d9f-461a-af5d-b30fd2ed8428-istiod-ca-cert\") pod \"fa482183-5d9f-461a-af5d-b30fd2ed8428\" (UID: \"fa482183-5d9f-461a-af5d-b30fd2ed8428\") "
Oct 30 22:11:01.066453 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.066426    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-workload-socket" pod "fa482183-5d9f-461a-af5d-b30fd2ed8428" (UID: "fa482183-5d9f-461a-af5d-b30fd2ed8428"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:01.067099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.066761    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-credential-socket" pod "fa482183-5d9f-461a-af5d-b30fd2ed8428" (UID: "fa482183-5d9f-461a-af5d-b30fd2ed8428"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:01.067099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.066843    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-data" pod "fa482183-5d9f-461a-af5d-b30fd2ed8428" (UID: "fa482183-5d9f-461a-af5d-b30fd2ed8428"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:01.067099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.067049    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.067099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.067069    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.067099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.067074    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa482183-5d9f-461a-af5d-b30fd2ed8428-istiod-ca-cert" pod "fa482183-5d9f-461a-af5d-b30fd2ed8428" (UID: "fa482183-5d9f-461a-af5d-b30fd2ed8428"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:01.067099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.067083    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.067487 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.067119    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.067487 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.067162    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.067487 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.067186    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.067487 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.067200    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.067487 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.067214    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.067487 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.067228    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.067487 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.067241    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzpw6\" (UniqueName: \"kubernetes.io/projected/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-kube-api-access-tzpw6\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.067487 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.067255    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.070103 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.070082    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:11:01.071085 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.071060    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-podinfo" pod "fa482183-5d9f-461a-af5d-b30fd2ed8428" (UID: "fa482183-5d9f-461a-af5d-b30fd2ed8428"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:01.071268 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.071245    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa482183-5d9f-461a-af5d-b30fd2ed8428-kube-api-access-b6zj6" pod "fa482183-5d9f-461a-af5d-b30fd2ed8428" (UID: "fa482183-5d9f-461a-af5d-b30fd2ed8428"). InnerVolumeSpecName "kube-api-access-b6zj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:01.071346 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.071276    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-token" pod "fa482183-5d9f-461a-af5d-b30fd2ed8428" (UID: "fa482183-5d9f-461a-af5d-b30fd2ed8428"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:01.072187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.072169    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-envoy" pod "fa482183-5d9f-461a-af5d-b30fd2ed8428" (UID: "fa482183-5d9f-461a-af5d-b30fd2ed8428"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:01.168719 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.168627    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e4c8521-f6ec-4d79-8946-045048bdad39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4c8521-f6ec-4d79-8946-045048bdad39-istiod-ca-cert\") pod \"3e4c8521-f6ec-4d79-8946-045048bdad39\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") "
Oct 30 22:11:01.168719 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.168681    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-credential-socket\") pod \"3e4c8521-f6ec-4d79-8946-045048bdad39\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") "
Oct 30 22:11:01.168974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.168753    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-envoy\") pod \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") "
Oct 30 22:11:01.168974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.168790    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-podinfo\") pod \"3e4c8521-f6ec-4d79-8946-045048bdad39\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") "
Oct 30 22:11:01.168974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.168828    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-token\" (UniqueName: \"kubernetes.io/projected/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-token\") pod \"3e4c8521-f6ec-4d79-8946-045048bdad39\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") "
Oct 30 22:11:01.168974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.168861    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-workload-socket\") pod \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") "
Oct 30 22:11:01.168974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.168905    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-workload-certs\") pod \"3e4c8521-f6ec-4d79-8946-045048bdad39\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") "
Oct 30 22:11:01.168974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.168956    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-workload-socket\") pod \"3e4c8521-f6ec-4d79-8946-045048bdad39\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") "
Oct 30 22:11:01.169316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.168996    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-data\") pod \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") "
Oct 30 22:11:01.169316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.169052    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e4c8521-f6ec-4d79-8946-045048bdad39-kube-api-access-tp7xz\" (UniqueName: \"kubernetes.io/projected/3e4c8521-f6ec-4d79-8946-045048bdad39-kube-api-access-tp7xz\") pod \"3e4c8521-f6ec-4d79-8946-045048bdad39\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") "
Oct 30 22:11:01.169316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.169059    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e4c8521-f6ec-4d79-8946-045048bdad39-istiod-ca-cert" pod "3e4c8521-f6ec-4d79-8946-045048bdad39" (UID: "3e4c8521-f6ec-4d79-8946-045048bdad39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:01.169316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.169100    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-credential-socket\") pod \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") "
Oct 30 22:11:01.169316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.169163    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-podinfo\") pod \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") "
Oct 30 22:11:01.169316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.169207    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-data\") pod \"3e4c8521-f6ec-4d79-8946-045048bdad39\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") "
Oct 30 22:11:01.169316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.169293    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istiod-ca-cert\") pod \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") "
Oct 30 22:11:01.169570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.169393    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-workload-certs\") pod \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") "
Oct 30 22:11:01.169570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.169415    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-workload-socket" pod "3e4c8521-f6ec-4d79-8946-045048bdad39" (UID: "3e4c8521-f6ec-4d79-8946-045048bdad39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:01.169570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.169476    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-token\" (UniqueName: \"kubernetes.io/projected/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-token\") pod \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") "
Oct 30 22:11:01.169570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.169550    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb761f4-75aa-481c-ad0a-0cfc782028ac-kube-api-access-tlzwd\" (UniqueName: \"kubernetes.io/projected/4bb761f4-75aa-481c-ad0a-0cfc782028ac-kube-api-access-tlzwd\") pod \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\" (UID: \"4bb761f4-75aa-481c-ad0a-0cfc782028ac\") "
Oct 30 22:11:01.169763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.169664    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-envoy\") pod \"3e4c8521-f6ec-4d79-8946-045048bdad39\" (UID: \"3e4c8521-f6ec-4d79-8946-045048bdad39\") "
Oct 30 22:11:01.169763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.169668    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-credential-socket" pod "3e4c8521-f6ec-4d79-8946-045048bdad39" (UID: "3e4c8521-f6ec-4d79-8946-045048bdad39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:01.170034 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.169897    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-data" pod "3e4c8521-f6ec-4d79-8946-045048bdad39" (UID: "3e4c8521-f6ec-4d79-8946-045048bdad39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:01.171855 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.171823    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-credential-socket" pod "4bb761f4-75aa-481c-ad0a-0cfc782028ac" (UID: "4bb761f4-75aa-481c-ad0a-0cfc782028ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:01.172077 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.172054    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-data" pod "4bb761f4-75aa-481c-ad0a-0cfc782028ac" (UID: "4bb761f4-75aa-481c-ad0a-0cfc782028ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:01.172358 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.172329    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-workload-certs" pod "4bb761f4-75aa-481c-ad0a-0cfc782028ac" (UID: "4bb761f4-75aa-481c-ad0a-0cfc782028ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:01.172460 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.172406    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-envoy" pod "4bb761f4-75aa-481c-ad0a-0cfc782028ac" (UID: "4bb761f4-75aa-481c-ad0a-0cfc782028ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:01.172460 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.172439    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.172572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.172458    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.172572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.172477    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4c8521-f6ec-4d79-8946-045048bdad39-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.172572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.172491    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.172572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.172506    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.172572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.172522    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.172572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.172535    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa482183-5d9f-461a-af5d-b30fd2ed8428-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.172572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.172550    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6zj6\" (UniqueName: \"kubernetes.io/projected/fa482183-5d9f-461a-af5d-b30fd2ed8428-kube-api-access-b6zj6\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.172572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.172565    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa482183-5d9f-461a-af5d-b30fd2ed8428-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.173011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.172639    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istiod-ca-cert" pod "4bb761f4-75aa-481c-ad0a-0cfc782028ac" (UID: "4bb761f4-75aa-481c-ad0a-0cfc782028ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:01.173011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.172779    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-workload-socket" pod "4bb761f4-75aa-481c-ad0a-0cfc782028ac" (UID: "4bb761f4-75aa-481c-ad0a-0cfc782028ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:01.173011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.172986    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-token" pod "3e4c8521-f6ec-4d79-8946-045048bdad39" (UID: "3e4c8521-f6ec-4d79-8946-045048bdad39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:01.173211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.173063    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-workload-certs" pod "3e4c8521-f6ec-4d79-8946-045048bdad39" (UID: "3e4c8521-f6ec-4d79-8946-045048bdad39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:01.173211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.173103    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4c8521-f6ec-4d79-8946-045048bdad39-kube-api-access-tp7xz" pod "3e4c8521-f6ec-4d79-8946-045048bdad39" (UID: "3e4c8521-f6ec-4d79-8946-045048bdad39"). InnerVolumeSpecName "kube-api-access-tp7xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:01.173445 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.173410    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-envoy" pod "3e4c8521-f6ec-4d79-8946-045048bdad39" (UID: "3e4c8521-f6ec-4d79-8946-045048bdad39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:01.175203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.175158    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-podinfo" pod "4bb761f4-75aa-481c-ad0a-0cfc782028ac" (UID: "4bb761f4-75aa-481c-ad0a-0cfc782028ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:01.175343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.175325    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb761f4-75aa-481c-ad0a-0cfc782028ac-kube-api-access-tlzwd" pod "4bb761f4-75aa-481c-ad0a-0cfc782028ac" (UID: "4bb761f4-75aa-481c-ad0a-0cfc782028ac"). InnerVolumeSpecName "kube-api-access-tlzwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:01.175397 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.175342    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-podinfo" pod "3e4c8521-f6ec-4d79-8946-045048bdad39" (UID: "3e4c8521-f6ec-4d79-8946-045048bdad39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:01.176467 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.176444    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-token" pod "4bb761f4-75aa-481c-ad0a-0cfc782028ac" (UID: "4bb761f4-75aa-481c-ad0a-0cfc782028ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:01.273655 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.273622    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.273655 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.273651    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.273655 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.273662    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp7xz\" (UniqueName: \"kubernetes.io/projected/3e4c8521-f6ec-4d79-8946-045048bdad39-kube-api-access-tp7xz\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.274011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.273679    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.274011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.273692    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.274011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.273701    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.274011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.273709    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.274011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.273720    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.274011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.273731    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlzwd\" (UniqueName: \"kubernetes.io/projected/4bb761f4-75aa-481c-ad0a-0cfc782028ac-kube-api-access-tlzwd\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.274011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.273739    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.274011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.273748    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.274011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.273756    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.274011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.273764    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4c8521-f6ec-4d79-8946-045048bdad39-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.274011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.273772    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bb761f4-75aa-481c-ad0a-0cfc782028ac-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:01.595275 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.595233    4553 generic.go:356] "Generic (PLEG): container finished" podID="fa482183-5d9f-461a-af5d-b30fd2ed8428" containerID="e9ba3d7bc6a24adbc52da5c485bd54a4d072889f45276618a26e740b76244f85" exitCode=0
Oct 30 22:11:01.595461 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.595295    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"
Oct 30 22:11:01.595461 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.595329    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp" event={"ID":"fa482183-5d9f-461a-af5d-b30fd2ed8428","Type":"ContainerDied","Data":"e9ba3d7bc6a24adbc52da5c485bd54a4d072889f45276618a26e740b76244f85"}
Oct 30 22:11:01.595461 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.595393    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp" event={"ID":"fa482183-5d9f-461a-af5d-b30fd2ed8428","Type":"ContainerDied","Data":"bef109eff1fea61484f26d34fed8026102b02d43d139095337498003412cb5e6"}
Oct 30 22:11:01.595461 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.595422    4553 scope.go:122] "RemoveContainer" containerID="e9ba3d7bc6a24adbc52da5c485bd54a4d072889f45276618a26e740b76244f85"
Oct 30 22:11:01.597432 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.597406    4553 generic.go:356] "Generic (PLEG): container finished" podID="4bb761f4-75aa-481c-ad0a-0cfc782028ac" containerID="509b5b63acc19168ffc8d8fd98093ce9f9c2462a18c2e495b4b143667c7ee5b7" exitCode=0
Oct 30 22:11:01.597559 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.597457    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"
Oct 30 22:11:01.597559 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.597532    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n" event={"ID":"4bb761f4-75aa-481c-ad0a-0cfc782028ac","Type":"ContainerDied","Data":"509b5b63acc19168ffc8d8fd98093ce9f9c2462a18c2e495b4b143667c7ee5b7"}
Oct 30 22:11:01.597721 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.597642    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n" event={"ID":"4bb761f4-75aa-481c-ad0a-0cfc782028ac","Type":"ContainerDied","Data":"8f6a8b7a9037bfef2101bd45d9bd2a9ca8d80843b973688655c0be6f4f82175e"}
Oct 30 22:11:01.599811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.599781    4553 generic.go:356] "Generic (PLEG): container finished" podID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" containerID="859ba0b5db24ba75490f7df78c49586274caf8f859d10d5c8d027b31d0bb8619" exitCode=0
Oct 30 22:11:01.599970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.599956    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"
Oct 30 22:11:01.600040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.599966    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b" event={"ID":"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3","Type":"ContainerDied","Data":"859ba0b5db24ba75490f7df78c49586274caf8f859d10d5c8d027b31d0bb8619"}
Oct 30 22:11:01.600040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.600003    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b" event={"ID":"2747809c-ab61-4e18-8cd3-e6bcfe0de5c3","Type":"ContainerDied","Data":"7a7b3673de3c66d9b08d900dfedf0f9823a1ddf205fd6b5b20d0416e2519edfe"}
Oct 30 22:11:01.601563 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.601541    4553 generic.go:356] "Generic (PLEG): container finished" podID="3e4c8521-f6ec-4d79-8946-045048bdad39" containerID="39f6245dd87a5e20de41fd6e8f0012d20d66d2b8f8159277760d2c423c4923d1" exitCode=0
Oct 30 22:11:01.601666 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.601582    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"
Oct 30 22:11:01.601666 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.601613    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr" event={"ID":"3e4c8521-f6ec-4d79-8946-045048bdad39","Type":"ContainerDied","Data":"39f6245dd87a5e20de41fd6e8f0012d20d66d2b8f8159277760d2c423c4923d1"}
Oct 30 22:11:01.601666 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.601662    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr" event={"ID":"3e4c8521-f6ec-4d79-8946-045048bdad39","Type":"ContainerDied","Data":"d5c610a0b3929cff23bfa1ee2fbd3022af156f2d2cb182d5d100b42ebaadfb42"}
Oct 30 22:11:01.608676 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.608657    4553 scope.go:122] "RemoveContainer" containerID="e9ba3d7bc6a24adbc52da5c485bd54a4d072889f45276618a26e740b76244f85"
Oct 30 22:11:01.608995 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:01.608977    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ba3d7bc6a24adbc52da5c485bd54a4d072889f45276618a26e740b76244f85\": container with ID starting with e9ba3d7bc6a24adbc52da5c485bd54a4d072889f45276618a26e740b76244f85 not found: ID does not exist" containerID="e9ba3d7bc6a24adbc52da5c485bd54a4d072889f45276618a26e740b76244f85"
Oct 30 22:11:01.609044 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.609004    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ba3d7bc6a24adbc52da5c485bd54a4d072889f45276618a26e740b76244f85"} err="failed to get container status \"e9ba3d7bc6a24adbc52da5c485bd54a4d072889f45276618a26e740b76244f85\": rpc error: code = NotFound desc = could not find container \"e9ba3d7bc6a24adbc52da5c485bd54a4d072889f45276618a26e740b76244f85\": container with ID starting with e9ba3d7bc6a24adbc52da5c485bd54a4d072889f45276618a26e740b76244f85 not found: ID does not exist"
Oct 30 22:11:01.609044 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.609023    4553 scope.go:122] "RemoveContainer" containerID="509b5b63acc19168ffc8d8fd98093ce9f9c2462a18c2e495b4b143667c7ee5b7"
Oct 30 22:11:01.618581 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.618563    4553 scope.go:122] "RemoveContainer" containerID="509b5b63acc19168ffc8d8fd98093ce9f9c2462a18c2e495b4b143667c7ee5b7"
Oct 30 22:11:01.618877 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:01.618852    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509b5b63acc19168ffc8d8fd98093ce9f9c2462a18c2e495b4b143667c7ee5b7\": container with ID starting with 509b5b63acc19168ffc8d8fd98093ce9f9c2462a18c2e495b4b143667c7ee5b7 not found: ID does not exist" containerID="509b5b63acc19168ffc8d8fd98093ce9f9c2462a18c2e495b4b143667c7ee5b7"
Oct 30 22:11:01.618925 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.618881    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509b5b63acc19168ffc8d8fd98093ce9f9c2462a18c2e495b4b143667c7ee5b7"} err="failed to get container status \"509b5b63acc19168ffc8d8fd98093ce9f9c2462a18c2e495b4b143667c7ee5b7\": rpc error: code = NotFound desc = could not find container \"509b5b63acc19168ffc8d8fd98093ce9f9c2462a18c2e495b4b143667c7ee5b7\": container with ID starting with 509b5b63acc19168ffc8d8fd98093ce9f9c2462a18c2e495b4b143667c7ee5b7 not found: ID does not exist"
Oct 30 22:11:01.618925 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.618900    4553 scope.go:122] "RemoveContainer" containerID="859ba0b5db24ba75490f7df78c49586274caf8f859d10d5c8d027b31d0bb8619"
Oct 30 22:11:01.628070 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.628048    4553 scope.go:122] "RemoveContainer" containerID="57b9223804e5c881c6c852272e96b288fd26968f286e3cf5e41f0687e6444dd7"
Oct 30 22:11:01.637210 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.637192    4553 scope.go:122] "RemoveContainer" containerID="859ba0b5db24ba75490f7df78c49586274caf8f859d10d5c8d027b31d0bb8619"
Oct 30 22:11:01.637513 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:01.637491    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"859ba0b5db24ba75490f7df78c49586274caf8f859d10d5c8d027b31d0bb8619\": container with ID starting with 859ba0b5db24ba75490f7df78c49586274caf8f859d10d5c8d027b31d0bb8619 not found: ID does not exist" containerID="859ba0b5db24ba75490f7df78c49586274caf8f859d10d5c8d027b31d0bb8619"
Oct 30 22:11:01.637573 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.637522    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859ba0b5db24ba75490f7df78c49586274caf8f859d10d5c8d027b31d0bb8619"} err="failed to get container status \"859ba0b5db24ba75490f7df78c49586274caf8f859d10d5c8d027b31d0bb8619\": rpc error: code = NotFound desc = could not find container \"859ba0b5db24ba75490f7df78c49586274caf8f859d10d5c8d027b31d0bb8619\": container with ID starting with 859ba0b5db24ba75490f7df78c49586274caf8f859d10d5c8d027b31d0bb8619 not found: ID does not exist"
Oct 30 22:11:01.637573 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.637543    4553 scope.go:122] "RemoveContainer" containerID="57b9223804e5c881c6c852272e96b288fd26968f286e3cf5e41f0687e6444dd7"
Oct 30 22:11:01.637803 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:01.637785    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b9223804e5c881c6c852272e96b288fd26968f286e3cf5e41f0687e6444dd7\": container with ID starting with 57b9223804e5c881c6c852272e96b288fd26968f286e3cf5e41f0687e6444dd7 not found: ID does not exist" containerID="57b9223804e5c881c6c852272e96b288fd26968f286e3cf5e41f0687e6444dd7"
Oct 30 22:11:01.637863 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.637814    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b9223804e5c881c6c852272e96b288fd26968f286e3cf5e41f0687e6444dd7"} err="failed to get container status \"57b9223804e5c881c6c852272e96b288fd26968f286e3cf5e41f0687e6444dd7\": rpc error: code = NotFound desc = could not find container \"57b9223804e5c881c6c852272e96b288fd26968f286e3cf5e41f0687e6444dd7\": container with ID starting with 57b9223804e5c881c6c852272e96b288fd26968f286e3cf5e41f0687e6444dd7 not found: ID does not exist"
Oct 30 22:11:01.637863 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.637839    4553 scope.go:122] "RemoveContainer" containerID="39f6245dd87a5e20de41fd6e8f0012d20d66d2b8f8159277760d2c423c4923d1"
Oct 30 22:11:01.650731 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.650708    4553 scope.go:122] "RemoveContainer" containerID="39f6245dd87a5e20de41fd6e8f0012d20d66d2b8f8159277760d2c423c4923d1"
Oct 30 22:11:01.651126 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:01.651103    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f6245dd87a5e20de41fd6e8f0012d20d66d2b8f8159277760d2c423c4923d1\": container with ID starting with 39f6245dd87a5e20de41fd6e8f0012d20d66d2b8f8159277760d2c423c4923d1 not found: ID does not exist" containerID="39f6245dd87a5e20de41fd6e8f0012d20d66d2b8f8159277760d2c423c4923d1"
Oct 30 22:11:01.651243 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.651151    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f6245dd87a5e20de41fd6e8f0012d20d66d2b8f8159277760d2c423c4923d1"} err="failed to get container status \"39f6245dd87a5e20de41fd6e8f0012d20d66d2b8f8159277760d2c423c4923d1\": rpc error: code = NotFound desc = could not find container \"39f6245dd87a5e20de41fd6e8f0012d20d66d2b8f8159277760d2c423c4923d1\": container with ID starting with 39f6245dd87a5e20de41fd6e8f0012d20d66d2b8f8159277760d2c423c4923d1 not found: ID does not exist"
Oct 30 22:11:01.764921 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.764817    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"]
Oct 30 22:11:01.768607 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.768564    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-pfr2n"]
Oct 30 22:11:01.879375 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.879337    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"]
Oct 30 22:11:01.882929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.882904    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-7fbtp"]
Oct 30 22:11:01.994033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.993992    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"]
Oct 30 22:11:01.998063 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:01.998031    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-z7bsr"]
Oct 30 22:11:02.109529 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:02.109494    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"]
Oct 30 22:11:02.113405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:02.113375    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-qqt5b"]
Oct 30 22:11:02.784422 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:02.784392    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"]
Oct 30 22:11:02.784623 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:02.784597    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh" podUID="8324c7c5-ac7a-439d-9f12-048a609e6ab3" containerName="maas-api" containerID="cri-o://8232d3a1cea90b3d8774a27be51a4794c70314959d53bfdb3138b06520c84fed" gracePeriod=30
Oct 30 22:11:03.140896 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.140868    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:11:03.284205 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.283734    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz" podUID="7ee10de5-d1f3-47dd-a0e8-a97e11381b1e" containerName="payload-pre-processing" containerID="cri-o://ef8e2da9f6270901e7b9896b0ca7272f44564d7972e05e6e38ffaa97cefcf1b5" gracePeriod=25
Oct 30 22:11:03.308855 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.308434    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-metrics-certs\") pod \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") "
Oct 30 22:11:03.308855 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.308544    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-maas-api-tls\") pod \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") "
Oct 30 22:11:03.308855 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.308583    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8324c7c5-ac7a-439d-9f12-048a609e6ab3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8324c7c5-ac7a-439d-9f12-048a609e6ab3-openshift-service-ca\") pod \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") "
Oct 30 22:11:03.308855 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.308636    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8324c7c5-ac7a-439d-9f12-048a609e6ab3-kube-api-access-mkdps\" (UniqueName: \"kubernetes.io/projected/8324c7c5-ac7a-439d-9f12-048a609e6ab3-kube-api-access-mkdps\") pod \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\" (UID: \"8324c7c5-ac7a-439d-9f12-048a609e6ab3\") "
Oct 30 22:11:03.311940 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.311834    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8324c7c5-ac7a-439d-9f12-048a609e6ab3-openshift-service-ca" pod "8324c7c5-ac7a-439d-9f12-048a609e6ab3" (UID: "8324c7c5-ac7a-439d-9f12-048a609e6ab3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:03.313692 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.313207    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8324c7c5-ac7a-439d-9f12-048a609e6ab3-kube-api-access-mkdps" pod "8324c7c5-ac7a-439d-9f12-048a609e6ab3" (UID: "8324c7c5-ac7a-439d-9f12-048a609e6ab3"). InnerVolumeSpecName "kube-api-access-mkdps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.316280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.316247    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-maas-api-tls" pod "8324c7c5-ac7a-439d-9f12-048a609e6ab3" (UID: "8324c7c5-ac7a-439d-9f12-048a609e6ab3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:03.317459 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.317436    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-metrics-certs" pod "8324c7c5-ac7a-439d-9f12-048a609e6ab3" (UID: "8324c7c5-ac7a-439d-9f12-048a609e6ab3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:03.427926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.426008    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.427926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.426046    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8324c7c5-ac7a-439d-9f12-048a609e6ab3-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.427926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.426063    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8324c7c5-ac7a-439d-9f12-048a609e6ab3-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.427926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.426078    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkdps\" (UniqueName: \"kubernetes.io/projected/8324c7c5-ac7a-439d-9f12-048a609e6ab3-kube-api-access-mkdps\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.464450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.464410    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" path="/var/lib/kubelet/pods/2747809c-ab61-4e18-8cd3-e6bcfe0de5c3/volumes"
Oct 30 22:11:03.465051 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.465031    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e4c8521-f6ec-4d79-8946-045048bdad39" path="/var/lib/kubelet/pods/3e4c8521-f6ec-4d79-8946-045048bdad39/volumes"
Oct 30 22:11:03.465597 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.465581    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb761f4-75aa-481c-ad0a-0cfc782028ac" path="/var/lib/kubelet/pods/4bb761f4-75aa-481c-ad0a-0cfc782028ac/volumes"
Oct 30 22:11:03.466385 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.466369    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa482183-5d9f-461a-af5d-b30fd2ed8428" path="/var/lib/kubelet/pods/fa482183-5d9f-461a-af5d-b30fd2ed8428/volumes"
Oct 30 22:11:03.606100 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.606036    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"
Oct 30 22:11:03.619652 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.619619    4553 generic.go:356] "Generic (PLEG): container finished" podID="8324c7c5-ac7a-439d-9f12-048a609e6ab3" containerID="8232d3a1cea90b3d8774a27be51a4794c70314959d53bfdb3138b06520c84fed" exitCode=0
Oct 30 22:11:03.619815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.619694    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"
Oct 30 22:11:03.619815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.619710    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh" event={"ID":"8324c7c5-ac7a-439d-9f12-048a609e6ab3","Type":"ContainerDied","Data":"8232d3a1cea90b3d8774a27be51a4794c70314959d53bfdb3138b06520c84fed"}
Oct 30 22:11:03.619815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.619777    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh" event={"ID":"8324c7c5-ac7a-439d-9f12-048a609e6ab3","Type":"ContainerDied","Data":"d7ed58c7435a04119653bc3bf9836530ce24054830f571d8f1f172a6a51a5976"}
Oct 30 22:11:03.619815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.619810    4553 scope.go:122] "RemoveContainer" containerID="8232d3a1cea90b3d8774a27be51a4794c70314959d53bfdb3138b06520c84fed"
Oct 30 22:11:03.621250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.621227    4553 generic.go:356] "Generic (PLEG): container finished" podID="7ee10de5-d1f3-47dd-a0e8-a97e11381b1e" containerID="ef8e2da9f6270901e7b9896b0ca7272f44564d7972e05e6e38ffaa97cefcf1b5" exitCode=0
Oct 30 22:11:03.621390 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.621274    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"
Oct 30 22:11:03.621469 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.621384    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz" event={"ID":"7ee10de5-d1f3-47dd-a0e8-a97e11381b1e","Type":"ContainerDied","Data":"ef8e2da9f6270901e7b9896b0ca7272f44564d7972e05e6e38ffaa97cefcf1b5"}
Oct 30 22:11:03.621469 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.621434    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz" event={"ID":"7ee10de5-d1f3-47dd-a0e8-a97e11381b1e","Type":"ContainerDied","Data":"c32f13292b6e883b3e5496c3aabf5cda741eebc884298d8d37fae2b4f9d0a5b8"}
Oct 30 22:11:03.631011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.630987    4553 scope.go:122] "RemoveContainer" containerID="8232d3a1cea90b3d8774a27be51a4794c70314959d53bfdb3138b06520c84fed"
Oct 30 22:11:03.631355 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:03.631338    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8232d3a1cea90b3d8774a27be51a4794c70314959d53bfdb3138b06520c84fed\": container with ID starting with 8232d3a1cea90b3d8774a27be51a4794c70314959d53bfdb3138b06520c84fed not found: ID does not exist" containerID="8232d3a1cea90b3d8774a27be51a4794c70314959d53bfdb3138b06520c84fed"
Oct 30 22:11:03.631430 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.631365    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8232d3a1cea90b3d8774a27be51a4794c70314959d53bfdb3138b06520c84fed"} err="failed to get container status \"8232d3a1cea90b3d8774a27be51a4794c70314959d53bfdb3138b06520c84fed\": rpc error: code = NotFound desc = could not find container \"8232d3a1cea90b3d8774a27be51a4794c70314959d53bfdb3138b06520c84fed\": container with ID starting with 8232d3a1cea90b3d8774a27be51a4794c70314959d53bfdb3138b06520c84fed not found: ID does not exist"
Oct 30 22:11:03.631430 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.631385    4553 scope.go:122] "RemoveContainer" containerID="ef8e2da9f6270901e7b9896b0ca7272f44564d7972e05e6e38ffaa97cefcf1b5"
Oct 30 22:11:03.640905 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.640887    4553 scope.go:122] "RemoveContainer" containerID="ef8e2da9f6270901e7b9896b0ca7272f44564d7972e05e6e38ffaa97cefcf1b5"
Oct 30 22:11:03.641243 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:03.641224    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8e2da9f6270901e7b9896b0ca7272f44564d7972e05e6e38ffaa97cefcf1b5\": container with ID starting with ef8e2da9f6270901e7b9896b0ca7272f44564d7972e05e6e38ffaa97cefcf1b5 not found: ID does not exist" containerID="ef8e2da9f6270901e7b9896b0ca7272f44564d7972e05e6e38ffaa97cefcf1b5"
Oct 30 22:11:03.641321 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.641253    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8e2da9f6270901e7b9896b0ca7272f44564d7972e05e6e38ffaa97cefcf1b5"} err="failed to get container status \"ef8e2da9f6270901e7b9896b0ca7272f44564d7972e05e6e38ffaa97cefcf1b5\": rpc error: code = NotFound desc = could not find container \"ef8e2da9f6270901e7b9896b0ca7272f44564d7972e05e6e38ffaa97cefcf1b5\": container with ID starting with ef8e2da9f6270901e7b9896b0ca7272f44564d7972e05e6e38ffaa97cefcf1b5 not found: ID does not exist"
Oct 30 22:11:03.729485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.729445    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee10de5-d1f3-47dd-a0e8-a97e11381b1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee10de5-d1f3-47dd-a0e8-a97e11381b1e-plugins-config-volume\") pod \"7ee10de5-d1f3-47dd-a0e8-a97e11381b1e\" (UID: \"7ee10de5-d1f3-47dd-a0e8-a97e11381b1e\") "
Oct 30 22:11:03.729701 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.729604    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee10de5-d1f3-47dd-a0e8-a97e11381b1e-kube-api-access-k6fq4\" (UniqueName: \"kubernetes.io/projected/7ee10de5-d1f3-47dd-a0e8-a97e11381b1e-kube-api-access-k6fq4\") pod \"7ee10de5-d1f3-47dd-a0e8-a97e11381b1e\" (UID: \"7ee10de5-d1f3-47dd-a0e8-a97e11381b1e\") "
Oct 30 22:11:03.729935 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.729889    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee10de5-d1f3-47dd-a0e8-a97e11381b1e-plugins-config-volume" pod "7ee10de5-d1f3-47dd-a0e8-a97e11381b1e" (UID: "7ee10de5-d1f3-47dd-a0e8-a97e11381b1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:03.730353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.730315    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee10de5-d1f3-47dd-a0e8-a97e11381b1e-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.733044 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.733017    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee10de5-d1f3-47dd-a0e8-a97e11381b1e-kube-api-access-k6fq4" pod "7ee10de5-d1f3-47dd-a0e8-a97e11381b1e" (UID: "7ee10de5-d1f3-47dd-a0e8-a97e11381b1e"). InnerVolumeSpecName "kube-api-access-k6fq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:03.870854 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.849676    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6fq4\" (UniqueName: \"kubernetes.io/projected/7ee10de5-d1f3-47dd-a0e8-a97e11381b1e-kube-api-access-k6fq4\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:03.907160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.903097    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"]
Oct 30 22:11:03.916167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.915029    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-hfxrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:11:03.916167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.915107    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj" podUID="20775493-c504-4316-9d6f-816952c83e9a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:11:03.927622 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:03.927591    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-wpxlh"]
Oct 30 22:11:04.058933 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:04.058903    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"]
Oct 30 22:11:04.061878 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:04.061857    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-595d567959-tn2pz"]
Oct 30 22:11:04.904429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:04.904389    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-krsdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:11:04.904831 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:04.904475    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:11:04.938397 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:04.938364    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tdqzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:11:04.938584 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:04.938437    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 22:11:05.188675 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.188375    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 22:11:05.282259 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.281641    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-token\" (UniqueName: \"kubernetes.io/projected/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-token\") pod \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") "
Oct 30 22:11:05.283283 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.282622    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-podinfo\") pod \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") "
Oct 30 22:11:05.283283 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.282687    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-credential-socket\") pod \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") "
Oct 30 22:11:05.283283 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.282752    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-envoy\") pod \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") "
Oct 30 22:11:05.283283 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.282835    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d66e97b1-d8b1-477a-a8f7-82466dbcd432-kube-api-access-l649t\" (UniqueName: \"kubernetes.io/projected/d66e97b1-d8b1-477a-a8f7-82466dbcd432-kube-api-access-l649t\") pod \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") "
Oct 30 22:11:05.283283 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.282905    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-workload-certs\") pod \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") "
Oct 30 22:11:05.283283 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.282931    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-data\") pod \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") "
Oct 30 22:11:05.283283 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.283003    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istiod-ca-cert\") pod \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") "
Oct 30 22:11:05.283283 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.283086    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-workload-socket\") pod \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\" (UID: \"d66e97b1-d8b1-477a-a8f7-82466dbcd432\") "
Oct 30 22:11:05.295196 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.286234    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-data" pod "d66e97b1-d8b1-477a-a8f7-82466dbcd432" (UID: "d66e97b1-d8b1-477a-a8f7-82466dbcd432"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.295196 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.294258    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-workload-certs" pod "d66e97b1-d8b1-477a-a8f7-82466dbcd432" (UID: "d66e97b1-d8b1-477a-a8f7-82466dbcd432"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.295196 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.294512    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-credential-socket" pod "d66e97b1-d8b1-477a-a8f7-82466dbcd432" (UID: "d66e97b1-d8b1-477a-a8f7-82466dbcd432"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.296248 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.295831    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-workload-socket" pod "d66e97b1-d8b1-477a-a8f7-82466dbcd432" (UID: "d66e97b1-d8b1-477a-a8f7-82466dbcd432"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.307199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.296668    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istiod-ca-cert" pod "d66e97b1-d8b1-477a-a8f7-82466dbcd432" (UID: "d66e97b1-d8b1-477a-a8f7-82466dbcd432"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:05.307199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.297552    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.307199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.297578    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.307199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.297593    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.307199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.297605    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.307199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.297618    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.327241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.313385    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66e97b1-d8b1-477a-a8f7-82466dbcd432-kube-api-access-l649t" pod "d66e97b1-d8b1-477a-a8f7-82466dbcd432" (UID: "d66e97b1-d8b1-477a-a8f7-82466dbcd432"). InnerVolumeSpecName "kube-api-access-l649t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:05.354852 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.352803    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-envoy" pod "d66e97b1-d8b1-477a-a8f7-82466dbcd432" (UID: "d66e97b1-d8b1-477a-a8f7-82466dbcd432"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.355675 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.355601    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-podinfo" pod "d66e97b1-d8b1-477a-a8f7-82466dbcd432" (UID: "d66e97b1-d8b1-477a-a8f7-82466dbcd432"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:05.359711 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.359659    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-token" pod "d66e97b1-d8b1-477a-a8f7-82466dbcd432" (UID: "d66e97b1-d8b1-477a-a8f7-82466dbcd432"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:05.409702 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.404571    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.409702 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.404610    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.409702 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.404626    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l649t\" (UniqueName: \"kubernetes.io/projected/d66e97b1-d8b1-477a-a8f7-82466dbcd432-kube-api-access-l649t\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.409702 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.404641    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d66e97b1-d8b1-477a-a8f7-82466dbcd432-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.412750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.412322    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee10de5-d1f3-47dd-a0e8-a97e11381b1e" path="/var/lib/kubelet/pods/7ee10de5-d1f3-47dd-a0e8-a97e11381b1e/volumes"
Oct 30 22:11:05.414311 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.414055    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8324c7c5-ac7a-439d-9f12-048a609e6ab3" path="/var/lib/kubelet/pods/8324c7c5-ac7a-439d-9f12-048a609e6ab3/volumes"
Oct 30 22:11:05.421406 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.421377    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 22:11:05.478836 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.478653    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 22:11:05.489034 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.489010    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 22:11:05.505441 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505415    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istiod-ca-cert\") pod \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") "
Oct 30 22:11:05.505605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505459    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-workload-certs\") pod \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") "
Oct 30 22:11:05.505605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505514    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-data\") pod \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") "
Oct 30 22:11:05.505605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505548    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-token\" (UniqueName: \"kubernetes.io/projected/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-token\") pod \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") "
Oct 30 22:11:05.505605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505598    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-data\") pod \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") "
Oct 30 22:11:05.505814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505627    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-envoy\") pod \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") "
Oct 30 22:11:05.505814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505682    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-kube-api-access-jsx87\" (UniqueName: \"kubernetes.io/projected/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-kube-api-access-jsx87\") pod \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") "
Oct 30 22:11:05.505814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505720    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20775493-c504-4316-9d6f-816952c83e9a-istio-token\" (UniqueName: \"kubernetes.io/projected/20775493-c504-4316-9d6f-816952c83e9a-istio-token\") pod \"20775493-c504-4316-9d6f-816952c83e9a\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") "
Oct 30 22:11:05.505814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505792    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-workload-certs\") pod \"20775493-c504-4316-9d6f-816952c83e9a\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") "
Oct 30 22:11:05.506016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505815    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-workload-certs" pod "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" (UID: "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.506016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505863    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-data" pod "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" (UID: "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.506016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505872    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istiod-ca-cert" pod "c2b6e606-c225-4dd2-94c0-e8bfe42ee109" (UID: "c2b6e606-c225-4dd2-94c0-e8bfe42ee109"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:05.506016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505819    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-data" pod "c2b6e606-c225-4dd2-94c0-e8bfe42ee109" (UID: "c2b6e606-c225-4dd2-94c0-e8bfe42ee109"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.506016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505886    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-workload-certs\") pod \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") "
Oct 30 22:11:05.506016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505932    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-istio-data\") pod \"20775493-c504-4316-9d6f-816952c83e9a\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") "
Oct 30 22:11:05.506016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.505985    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-istio-envoy\") pod \"20775493-c504-4316-9d6f-816952c83e9a\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") "
Oct 30 22:11:05.506413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506027    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-credential-socket\") pod \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") "
Oct 30 22:11:05.506413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506061    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20775493-c504-4316-9d6f-816952c83e9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20775493-c504-4316-9d6f-816952c83e9a-istiod-ca-cert\") pod \"20775493-c504-4316-9d6f-816952c83e9a\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") "
Oct 30 22:11:05.506413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506089    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-workload-socket\") pod \"20775493-c504-4316-9d6f-816952c83e9a\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") "
Oct 30 22:11:05.506413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506111    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-workload-certs" pod "20775493-c504-4316-9d6f-816952c83e9a" (UID: "20775493-c504-4316-9d6f-816952c83e9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.506413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506122    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-workload-certs" pod "c2b6e606-c225-4dd2-94c0-e8bfe42ee109" (UID: "c2b6e606-c225-4dd2-94c0-e8bfe42ee109"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.506413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506202    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istiod-ca-cert\") pod \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") "
Oct 30 22:11:05.506413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506234    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-envoy\") pod \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") "
Oct 30 22:11:05.506413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506274    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-podinfo\") pod \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") "
Oct 30 22:11:05.506413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506350    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-kube-api-access-7vnlv\" (UniqueName: \"kubernetes.io/projected/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-kube-api-access-7vnlv\") pod \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") "
Oct 30 22:11:05.506413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506388    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-podinfo\") pod \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") "
Oct 30 22:11:05.506981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506418    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-credential-socket\") pod \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") "
Oct 30 22:11:05.506981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506453    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20775493-c504-4316-9d6f-816952c83e9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20775493-c504-4316-9d6f-816952c83e9a-istio-podinfo\") pod \"20775493-c504-4316-9d6f-816952c83e9a\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") "
Oct 30 22:11:05.506981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506491    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20775493-c504-4316-9d6f-816952c83e9a-kube-api-access-rflch\" (UniqueName: \"kubernetes.io/projected/20775493-c504-4316-9d6f-816952c83e9a-kube-api-access-rflch\") pod \"20775493-c504-4316-9d6f-816952c83e9a\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") "
Oct 30 22:11:05.506981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506523    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-credential-socket\") pod \"20775493-c504-4316-9d6f-816952c83e9a\" (UID: \"20775493-c504-4316-9d6f-816952c83e9a\") "
Oct 30 22:11:05.506981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506690    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-workload-socket\") pod \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\" (UID: \"c2b6e606-c225-4dd2-94c0-e8bfe42ee109\") "
Oct 30 22:11:05.506981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506757    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-workload-socket\") pod \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") "
Oct 30 22:11:05.506981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506805    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-token\" (UniqueName: \"kubernetes.io/projected/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-token\") pod \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\" (UID: \"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7\") "
Oct 30 22:11:05.507666 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.507642    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.507904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.507673    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.507904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.507688    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.507904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.507700    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.507904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.507715    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.507904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.507729    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.508827 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506385    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-credential-socket" pod "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" (UID: "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.508915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506443    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20775493-c504-4316-9d6f-816952c83e9a-istiod-ca-cert" pod "20775493-c504-4316-9d6f-816952c83e9a" (UID: "20775493-c504-4316-9d6f-816952c83e9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:05.508915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506565    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-istio-data" pod "20775493-c504-4316-9d6f-816952c83e9a" (UID: "20775493-c504-4316-9d6f-816952c83e9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.508915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.506845    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istiod-ca-cert" pod "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" (UID: "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:05.508915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.507053    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-workload-socket" pod "20775493-c504-4316-9d6f-816952c83e9a" (UID: "20775493-c504-4316-9d6f-816952c83e9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.508915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.508784    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-workload-socket" pod "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" (UID: "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.509244 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.508932    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-workload-socket" pod "c2b6e606-c225-4dd2-94c0-e8bfe42ee109" (UID: "c2b6e606-c225-4dd2-94c0-e8bfe42ee109"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.509244 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.509128    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-credential-socket" pod "c2b6e606-c225-4dd2-94c0-e8bfe42ee109" (UID: "c2b6e606-c225-4dd2-94c0-e8bfe42ee109"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.509244 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.509124    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-kube-api-access-jsx87" pod "c2b6e606-c225-4dd2-94c0-e8bfe42ee109" (UID: "c2b6e606-c225-4dd2-94c0-e8bfe42ee109"). InnerVolumeSpecName "kube-api-access-jsx87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:05.509244 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.509189    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-credential-socket" pod "20775493-c504-4316-9d6f-816952c83e9a" (UID: "20775493-c504-4316-9d6f-816952c83e9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.510340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.510313    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20775493-c504-4316-9d6f-816952c83e9a-istio-token" pod "20775493-c504-4316-9d6f-816952c83e9a" (UID: "20775493-c504-4316-9d6f-816952c83e9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:05.510534 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.510500    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-podinfo" pod "c2b6e606-c225-4dd2-94c0-e8bfe42ee109" (UID: "c2b6e606-c225-4dd2-94c0-e8bfe42ee109"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:05.510654 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.510629    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20775493-c504-4316-9d6f-816952c83e9a-istio-podinfo" pod "20775493-c504-4316-9d6f-816952c83e9a" (UID: "20775493-c504-4316-9d6f-816952c83e9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:05.510721 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.510685    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20775493-c504-4316-9d6f-816952c83e9a-kube-api-access-rflch" pod "20775493-c504-4316-9d6f-816952c83e9a" (UID: "20775493-c504-4316-9d6f-816952c83e9a"). InnerVolumeSpecName "kube-api-access-rflch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:05.510831 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.510813    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-token" pod "c2b6e606-c225-4dd2-94c0-e8bfe42ee109" (UID: "c2b6e606-c225-4dd2-94c0-e8bfe42ee109"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:05.510903 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.510856    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-envoy" pod "c2b6e606-c225-4dd2-94c0-e8bfe42ee109" (UID: "c2b6e606-c225-4dd2-94c0-e8bfe42ee109"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.511477 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.511450    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-istio-envoy" pod "20775493-c504-4316-9d6f-816952c83e9a" (UID: "20775493-c504-4316-9d6f-816952c83e9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.511477 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.511460    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-kube-api-access-7vnlv" pod "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" (UID: "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7"). InnerVolumeSpecName "kube-api-access-7vnlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:05.511761 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.511748    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-podinfo" pod "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" (UID: "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:05.512263 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.512250    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-token" pod "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" (UID: "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:05.512583 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.512562    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-envoy" pod "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" (UID: "abc77e33-55c0-4f13-8ff1-3ab8706e4ec7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:05.608766 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608727    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.608766 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608755    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.608766 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608764    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20775493-c504-4316-9d6f-816952c83e9a-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608779    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608791    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608800    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608809    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608819    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vnlv\" (UniqueName: \"kubernetes.io/projected/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-kube-api-access-7vnlv\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608828    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608836    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608844    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20775493-c504-4316-9d6f-816952c83e9a-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608853    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rflch\" (UniqueName: \"kubernetes.io/projected/20775493-c504-4316-9d6f-816952c83e9a-kube-api-access-rflch\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608862    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608871    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608878    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608886    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608894    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608903    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608911    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsx87\" (UniqueName: \"kubernetes.io/projected/c2b6e606-c225-4dd2-94c0-e8bfe42ee109-kube-api-access-jsx87\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608919    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20775493-c504-4316-9d6f-816952c83e9a-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.609003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.608926    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20775493-c504-4316-9d6f-816952c83e9a-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:05.640516 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.640478    4553 generic.go:356] "Generic (PLEG): container finished" podID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerID="a49f50aa9f216766b38da6c11287d01bfb2b90a7750f943277d496726956cae4" exitCode=0
Oct 30 22:11:05.640716 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.640530    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"
Oct 30 22:11:05.640716 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.640560    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" event={"ID":"d66e97b1-d8b1-477a-a8f7-82466dbcd432","Type":"ContainerDied","Data":"a49f50aa9f216766b38da6c11287d01bfb2b90a7750f943277d496726956cae4"}
Oct 30 22:11:05.640716 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.640621    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj" event={"ID":"d66e97b1-d8b1-477a-a8f7-82466dbcd432","Type":"ContainerDied","Data":"e920a1ab33d93c3c0db23615dad43d2f2ab1f70ec3b0e44f494ecd2c97169d2a"}
Oct 30 22:11:05.640716 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.640687    4553 scope.go:122] "RemoveContainer" containerID="a49f50aa9f216766b38da6c11287d01bfb2b90a7750f943277d496726956cae4"
Oct 30 22:11:05.642088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.642065    4553 generic.go:356] "Generic (PLEG): container finished" podID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerID="8e8a20b3f6e0915714ac7b35c7bfe1792148ae612743285943093523989f819a" exitCode=0
Oct 30 22:11:05.642202 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.642113    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"
Oct 30 22:11:05.642277 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.642185    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" event={"ID":"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7","Type":"ContainerDied","Data":"8e8a20b3f6e0915714ac7b35c7bfe1792148ae612743285943093523989f819a"}
Oct 30 22:11:05.642277 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.642262    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr" event={"ID":"abc77e33-55c0-4f13-8ff1-3ab8706e4ec7","Type":"ContainerDied","Data":"85ea78ec4d9adc178301b0da1033f2e308fde35ce36a71ad349529152032d5aa"}
Oct 30 22:11:05.643838 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.643813    4553 generic.go:356] "Generic (PLEG): container finished" podID="c2b6e606-c225-4dd2-94c0-e8bfe42ee109" containerID="8a10b957163558bea4ea4fb9711fcbd115b449a585cfc38f9b5ad1fd658e2d27" exitCode=0
Oct 30 22:11:05.643920 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.643851    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"
Oct 30 22:11:05.643974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.643959    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z" event={"ID":"c2b6e606-c225-4dd2-94c0-e8bfe42ee109","Type":"ContainerDied","Data":"8a10b957163558bea4ea4fb9711fcbd115b449a585cfc38f9b5ad1fd658e2d27"}
Oct 30 22:11:05.644018 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.643996    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z" event={"ID":"c2b6e606-c225-4dd2-94c0-e8bfe42ee109","Type":"ContainerDied","Data":"c8782dae5fa3f5fbd3bf752bc3a6f8eb295e46f3b5f6b0880d983d241f4dd674"}
Oct 30 22:11:05.652315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.652241    4553 generic.go:356] "Generic (PLEG): container finished" podID="20775493-c504-4316-9d6f-816952c83e9a" containerID="1ba57b2b607d4a4ec2a6c0a409d6c24898fcc8a932d16d197dd41e8999182789" exitCode=0
Oct 30 22:11:05.652315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.652288    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"
Oct 30 22:11:05.652483 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.652327    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj" event={"ID":"20775493-c504-4316-9d6f-816952c83e9a","Type":"ContainerDied","Data":"1ba57b2b607d4a4ec2a6c0a409d6c24898fcc8a932d16d197dd41e8999182789"}
Oct 30 22:11:05.652483 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.652380    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj" event={"ID":"20775493-c504-4316-9d6f-816952c83e9a","Type":"ContainerDied","Data":"b81d16b91da0c075c657a9873af7d26b6ceeb862183a17b22d5ffec00704f210"}
Oct 30 22:11:05.655512 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.655466    4553 scope.go:122] "RemoveContainer" containerID="a49f50aa9f216766b38da6c11287d01bfb2b90a7750f943277d496726956cae4"
Oct 30 22:11:05.656065 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:05.656038    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49f50aa9f216766b38da6c11287d01bfb2b90a7750f943277d496726956cae4\": container with ID starting with a49f50aa9f216766b38da6c11287d01bfb2b90a7750f943277d496726956cae4 not found: ID does not exist" containerID="a49f50aa9f216766b38da6c11287d01bfb2b90a7750f943277d496726956cae4"
Oct 30 22:11:05.656168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.656076    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49f50aa9f216766b38da6c11287d01bfb2b90a7750f943277d496726956cae4"} err="failed to get container status \"a49f50aa9f216766b38da6c11287d01bfb2b90a7750f943277d496726956cae4\": rpc error: code = NotFound desc = could not find container \"a49f50aa9f216766b38da6c11287d01bfb2b90a7750f943277d496726956cae4\": container with ID starting with a49f50aa9f216766b38da6c11287d01bfb2b90a7750f943277d496726956cae4 not found: ID does not exist"
Oct 30 22:11:05.656168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.656099    4553 scope.go:122] "RemoveContainer" containerID="8e8a20b3f6e0915714ac7b35c7bfe1792148ae612743285943093523989f819a"
Oct 30 22:11:05.680530 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.680500    4553 scope.go:122] "RemoveContainer" containerID="8e8a20b3f6e0915714ac7b35c7bfe1792148ae612743285943093523989f819a"
Oct 30 22:11:05.680998 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:05.680974    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8a20b3f6e0915714ac7b35c7bfe1792148ae612743285943093523989f819a\": container with ID starting with 8e8a20b3f6e0915714ac7b35c7bfe1792148ae612743285943093523989f819a not found: ID does not exist" containerID="8e8a20b3f6e0915714ac7b35c7bfe1792148ae612743285943093523989f819a"
Oct 30 22:11:05.681154 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.681009    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8a20b3f6e0915714ac7b35c7bfe1792148ae612743285943093523989f819a"} err="failed to get container status \"8e8a20b3f6e0915714ac7b35c7bfe1792148ae612743285943093523989f819a\": rpc error: code = NotFound desc = could not find container \"8e8a20b3f6e0915714ac7b35c7bfe1792148ae612743285943093523989f819a\": container with ID starting with 8e8a20b3f6e0915714ac7b35c7bfe1792148ae612743285943093523989f819a not found: ID does not exist"
Oct 30 22:11:05.681154 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.681032    4553 scope.go:122] "RemoveContainer" containerID="8a10b957163558bea4ea4fb9711fcbd115b449a585cfc38f9b5ad1fd658e2d27"
Oct 30 22:11:05.694306 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.694279    4553 scope.go:122] "RemoveContainer" containerID="8a10b957163558bea4ea4fb9711fcbd115b449a585cfc38f9b5ad1fd658e2d27"
Oct 30 22:11:05.694693 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:05.694667    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a10b957163558bea4ea4fb9711fcbd115b449a585cfc38f9b5ad1fd658e2d27\": container with ID starting with 8a10b957163558bea4ea4fb9711fcbd115b449a585cfc38f9b5ad1fd658e2d27 not found: ID does not exist" containerID="8a10b957163558bea4ea4fb9711fcbd115b449a585cfc38f9b5ad1fd658e2d27"
Oct 30 22:11:05.694767 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.694706    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a10b957163558bea4ea4fb9711fcbd115b449a585cfc38f9b5ad1fd658e2d27"} err="failed to get container status \"8a10b957163558bea4ea4fb9711fcbd115b449a585cfc38f9b5ad1fd658e2d27\": rpc error: code = NotFound desc = could not find container \"8a10b957163558bea4ea4fb9711fcbd115b449a585cfc38f9b5ad1fd658e2d27\": container with ID starting with 8a10b957163558bea4ea4fb9711fcbd115b449a585cfc38f9b5ad1fd658e2d27 not found: ID does not exist"
Oct 30 22:11:05.694767 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.694731    4553 scope.go:122] "RemoveContainer" containerID="1ba57b2b607d4a4ec2a6c0a409d6c24898fcc8a932d16d197dd41e8999182789"
Oct 30 22:11:05.706731 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.706704    4553 scope.go:122] "RemoveContainer" containerID="1ba57b2b607d4a4ec2a6c0a409d6c24898fcc8a932d16d197dd41e8999182789"
Oct 30 22:11:05.707091 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:05.707069    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba57b2b607d4a4ec2a6c0a409d6c24898fcc8a932d16d197dd41e8999182789\": container with ID starting with 1ba57b2b607d4a4ec2a6c0a409d6c24898fcc8a932d16d197dd41e8999182789 not found: ID does not exist" containerID="1ba57b2b607d4a4ec2a6c0a409d6c24898fcc8a932d16d197dd41e8999182789"
Oct 30 22:11:05.707285 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.707105    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba57b2b607d4a4ec2a6c0a409d6c24898fcc8a932d16d197dd41e8999182789"} err="failed to get container status \"1ba57b2b607d4a4ec2a6c0a409d6c24898fcc8a932d16d197dd41e8999182789\": rpc error: code = NotFound desc = could not find container \"1ba57b2b607d4a4ec2a6c0a409d6c24898fcc8a932d16d197dd41e8999182789\": container with ID starting with 1ba57b2b607d4a4ec2a6c0a409d6c24898fcc8a932d16d197dd41e8999182789 not found: ID does not exist"
Oct 30 22:11:05.922083 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.922044    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"]
Oct 30 22:11:05.925824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:05.925797    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-hfxrj"]
Oct 30 22:11:06.037734 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:06.037698    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"]
Oct 30 22:11:06.040682 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:06.040656    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-j4c2z"]
Oct 30 22:11:06.151079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:06.151037    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"]
Oct 30 22:11:06.155963 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:06.155870    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tdqzj"]
Oct 30 22:11:06.267936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:06.267898    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"]
Oct 30 22:11:06.271310 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:06.271286    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-krsdr"]
Oct 30 22:11:07.394214 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:07.394170    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20775493-c504-4316-9d6f-816952c83e9a" path="/var/lib/kubelet/pods/20775493-c504-4316-9d6f-816952c83e9a/volumes"
Oct 30 22:11:07.394890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:07.394870    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" path="/var/lib/kubelet/pods/abc77e33-55c0-4f13-8ff1-3ab8706e4ec7/volumes"
Oct 30 22:11:07.395420 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:07.395401    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b6e606-c225-4dd2-94c0-e8bfe42ee109" path="/var/lib/kubelet/pods/c2b6e606-c225-4dd2-94c0-e8bfe42ee109/volumes"
Oct 30 22:11:07.396241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:07.396221    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" path="/var/lib/kubelet/pods/d66e97b1-d8b1-477a-a8f7-82466dbcd432/volumes"
Oct 30 22:11:09.245293 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.245259    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"]
Oct 30 22:11:09.245803 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.245486    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm" podUID="9e4d1744-6b44-46ae-8e36-0f3584c10ed6" containerName="maas-api" containerID="cri-o://c427a2387ea342a70d8e017e05a5751dfd3d78d886ea0d8b954413cdb1dbfd32" gracePeriod=30
Oct 30 22:11:09.583497 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.583470    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:11:09.674171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.666754    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-maas-api-tls\") pod \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") "
Oct 30 22:11:09.674171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.671651    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-kube-api-access-86zpb\" (UniqueName: \"kubernetes.io/projected/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-kube-api-access-86zpb\") pod \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") "
Oct 30 22:11:09.674171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.671704    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-openshift-service-ca\") pod \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") "
Oct 30 22:11:09.674171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.671989    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-metrics-certs\") pod \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\" (UID: \"9e4d1744-6b44-46ae-8e36-0f3584c10ed6\") "
Oct 30 22:11:09.681480 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.681372    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-openshift-service-ca" pod "9e4d1744-6b44-46ae-8e36-0f3584c10ed6" (UID: "9e4d1744-6b44-46ae-8e36-0f3584c10ed6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:09.722256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.704961    4553 generic.go:356] "Generic (PLEG): container finished" podID="9e4d1744-6b44-46ae-8e36-0f3584c10ed6" containerID="c427a2387ea342a70d8e017e05a5751dfd3d78d886ea0d8b954413cdb1dbfd32" exitCode=0
Oct 30 22:11:09.722256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.705161    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm" event={"ID":"9e4d1744-6b44-46ae-8e36-0f3584c10ed6","Type":"ContainerDied","Data":"c427a2387ea342a70d8e017e05a5751dfd3d78d886ea0d8b954413cdb1dbfd32"}
Oct 30 22:11:09.722256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.705222    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm" event={"ID":"9e4d1744-6b44-46ae-8e36-0f3584c10ed6","Type":"ContainerDied","Data":"070bbd2ff6d9ffcb8463497dafb9246a556322ffa8b175ed1cdb9ec05cf66ebb"}
Oct 30 22:11:09.722256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.705250    4553 scope.go:122] "RemoveContainer" containerID="c427a2387ea342a70d8e017e05a5751dfd3d78d886ea0d8b954413cdb1dbfd32"
Oct 30 22:11:09.722256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.705489    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"
Oct 30 22:11:09.722256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.720355    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-maas-api-tls" pod "9e4d1744-6b44-46ae-8e36-0f3584c10ed6" (UID: "9e4d1744-6b44-46ae-8e36-0f3584c10ed6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:09.746637 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.746598    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-metrics-certs" pod "9e4d1744-6b44-46ae-8e36-0f3584c10ed6" (UID: "9e4d1744-6b44-46ae-8e36-0f3584c10ed6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:09.746796 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.746653    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-kube-api-access-86zpb" pod "9e4d1744-6b44-46ae-8e36-0f3584c10ed6" (UID: "9e4d1744-6b44-46ae-8e36-0f3584c10ed6"). InnerVolumeSpecName "kube-api-access-86zpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:09.750031 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.750007    4553 scope.go:122] "RemoveContainer" containerID="c427a2387ea342a70d8e017e05a5751dfd3d78d886ea0d8b954413cdb1dbfd32"
Oct 30 22:11:09.750413 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:09.750394    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c427a2387ea342a70d8e017e05a5751dfd3d78d886ea0d8b954413cdb1dbfd32\": container with ID starting with c427a2387ea342a70d8e017e05a5751dfd3d78d886ea0d8b954413cdb1dbfd32 not found: ID does not exist" containerID="c427a2387ea342a70d8e017e05a5751dfd3d78d886ea0d8b954413cdb1dbfd32"
Oct 30 22:11:09.750483 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.750424    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c427a2387ea342a70d8e017e05a5751dfd3d78d886ea0d8b954413cdb1dbfd32"} err="failed to get container status \"c427a2387ea342a70d8e017e05a5751dfd3d78d886ea0d8b954413cdb1dbfd32\": rpc error: code = NotFound desc = could not find container \"c427a2387ea342a70d8e017e05a5751dfd3d78d886ea0d8b954413cdb1dbfd32\": container with ID starting with c427a2387ea342a70d8e017e05a5751dfd3d78d886ea0d8b954413cdb1dbfd32 not found: ID does not exist"
Oct 30 22:11:09.781062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.781026    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:09.781062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.781061    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86zpb\" (UniqueName: \"kubernetes.io/projected/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-kube-api-access-86zpb\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:09.781267 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.781077    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:09.781267 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:09.781093    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e4d1744-6b44-46ae-8e36-0f3584c10ed6-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:10.133314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:10.133280    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"]
Oct 30 22:11:10.183781 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:10.183741    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ac4a7-5b98444d97-q8czm"]
Oct 30 22:11:11.393555 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:11.393525    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e4d1744-6b44-46ae-8e36-0f3584c10ed6" path="/var/lib/kubelet/pods/9e4d1744-6b44-46ae-8e36-0f3584c10ed6/volumes"
Oct 30 22:11:25.882890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:25.882857    4553 generic.go:356] "Generic (PLEG): container finished" podID="cae0b720-525f-43fc-a94b-42e079ce85b4" containerID="81d22899ad6575a630c03418d4ce4d82410837e40ed6c76cb1900ccfbd55e7ff" exitCode=0
Oct 30 22:11:25.883382 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:25.882927    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq" event={"ID":"cae0b720-525f-43fc-a94b-42e079ce85b4","Type":"ContainerDied","Data":"81d22899ad6575a630c03418d4ce4d82410837e40ed6c76cb1900ccfbd55e7ff"}
Oct 30 22:11:25.883429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:25.883409    4553 scope.go:122] "RemoveContainer" containerID="81d22899ad6575a630c03418d4ce4d82410837e40ed6c76cb1900ccfbd55e7ff"
Oct 30 22:11:25.885175 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:25.885146    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:11:25.885245 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:25.885219    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:11:25.885281 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:25.885251    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:11:26.952874 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:26.952830    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq" event={"ID":"cae0b720-525f-43fc-a94b-42e079ce85b4","Type":"ContainerStarted","Data":"3db1554d97fd3c808a50fc26d55128136b9c835a49789fad86f73b18d046d0ca"}
Oct 30 22:11:27.133501 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:27.133458    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:11:27.134600 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:27.134574    4553 patch_prober.go:29] interesting pod/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:11:27.134723 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:27.134638    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq" podUID="cae0b720-525f-43fc-a94b-42e079ce85b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 22:11:28.133704 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:28.133659    4553 patch_prober.go:29] interesting pod/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:11:28.134303 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:28.133730    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq" podUID="cae0b720-525f-43fc-a94b-42e079ce85b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 22:11:28.203164 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:28.203116    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:11:29.133623 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:29.133584    4553 patch_prober.go:29] interesting pod/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:11:29.133815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:29.133652    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq" podUID="cae0b720-525f-43fc-a94b-42e079ce85b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 22:11:29.342055 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:29.342024    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"]
Oct 30 22:11:29.342336 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:29.342292    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s" podUID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerName="istio-proxy" containerID="cri-o://0be1c22599e8a6fb471786b544b570e4ab720231fe805d7f79a5ecd126ee4666" gracePeriod=30
Oct 30 22:11:29.480293 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:29.452427    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:11:29.575396 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:29.575353    4553 patch_prober.go:29] interesting pod/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:11:29.575573 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:29.575429    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s" podUID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:11:30.134337 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:30.132613    4553 patch_prober.go:29] interesting pod/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:11:30.134337 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:30.132681    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq" podUID="cae0b720-525f-43fc-a94b-42e079ce85b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 22:11:31.131162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:31.130645    4553 patch_prober.go:29] interesting pod/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:11:31.131162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:31.130710    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq" podUID="cae0b720-525f-43fc-a94b-42e079ce85b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 22:11:32.130397 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:32.130352    4553 patch_prober.go:29] interesting pod/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:11:32.130802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:32.130418    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq" podUID="cae0b720-525f-43fc-a94b-42e079ce85b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 22:11:33.134830 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:33.134794    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:11:33.135940 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:33.135917    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:11:34.615991 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.615922    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:11:34.785512 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.785475    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-data\") pod \"447fed40-9a8d-4a48-88a3-c9028d7792d5\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") "
Oct 30 22:11:34.785675 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.785548    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-credential-socket\") pod \"447fed40-9a8d-4a48-88a3-c9028d7792d5\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") "
Oct 30 22:11:34.785675 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.785572    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/447fed40-9a8d-4a48-88a3-c9028d7792d5-kube-api-access-x78qb\" (UniqueName: \"kubernetes.io/projected/447fed40-9a8d-4a48-88a3-c9028d7792d5-kube-api-access-x78qb\") pod \"447fed40-9a8d-4a48-88a3-c9028d7792d5\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") "
Oct 30 22:11:34.785675 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.785618    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-podinfo\") pod \"447fed40-9a8d-4a48-88a3-c9028d7792d5\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") "
Oct 30 22:11:34.785675 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.785652    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-workload-socket\") pod \"447fed40-9a8d-4a48-88a3-c9028d7792d5\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") "
Oct 30 22:11:34.785913 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.785701    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-envoy\") pod \"447fed40-9a8d-4a48-88a3-c9028d7792d5\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") "
Oct 30 22:11:34.785913 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.785794    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-token\" (UniqueName: \"kubernetes.io/projected/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-token\") pod \"447fed40-9a8d-4a48-88a3-c9028d7792d5\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") "
Oct 30 22:11:34.785913 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.785810    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-data" pod "447fed40-9a8d-4a48-88a3-c9028d7792d5" (UID: "447fed40-9a8d-4a48-88a3-c9028d7792d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:34.785913 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.785825    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-credential-socket" pod "447fed40-9a8d-4a48-88a3-c9028d7792d5" (UID: "447fed40-9a8d-4a48-88a3-c9028d7792d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:34.785913 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.785907    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-workload-certs\") pod \"447fed40-9a8d-4a48-88a3-c9028d7792d5\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") "
Oct 30 22:11:34.786295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.785958    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/447fed40-9a8d-4a48-88a3-c9028d7792d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447fed40-9a8d-4a48-88a3-c9028d7792d5-istiod-ca-cert\") pod \"447fed40-9a8d-4a48-88a3-c9028d7792d5\" (UID: \"447fed40-9a8d-4a48-88a3-c9028d7792d5\") "
Oct 30 22:11:34.786295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.786034    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-workload-socket" pod "447fed40-9a8d-4a48-88a3-c9028d7792d5" (UID: "447fed40-9a8d-4a48-88a3-c9028d7792d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:34.786295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.786119    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-workload-certs" pod "447fed40-9a8d-4a48-88a3-c9028d7792d5" (UID: "447fed40-9a8d-4a48-88a3-c9028d7792d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:34.786458 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.786345    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/447fed40-9a8d-4a48-88a3-c9028d7792d5-istiod-ca-cert" pod "447fed40-9a8d-4a48-88a3-c9028d7792d5" (UID: "447fed40-9a8d-4a48-88a3-c9028d7792d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:34.787300 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.787274    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:34.787422 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.787304    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:34.787422 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.787321    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/447fed40-9a8d-4a48-88a3-c9028d7792d5-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:34.787422 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.787336    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:34.787422 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.787351    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:34.788184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.788156    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-podinfo" pod "447fed40-9a8d-4a48-88a3-c9028d7792d5" (UID: "447fed40-9a8d-4a48-88a3-c9028d7792d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:34.788184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.788176    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-envoy" pod "447fed40-9a8d-4a48-88a3-c9028d7792d5" (UID: "447fed40-9a8d-4a48-88a3-c9028d7792d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:34.788350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.788210    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447fed40-9a8d-4a48-88a3-c9028d7792d5-kube-api-access-x78qb" pod "447fed40-9a8d-4a48-88a3-c9028d7792d5" (UID: "447fed40-9a8d-4a48-88a3-c9028d7792d5"). InnerVolumeSpecName "kube-api-access-x78qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:34.788350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.788270    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-token" pod "447fed40-9a8d-4a48-88a3-c9028d7792d5" (UID: "447fed40-9a8d-4a48-88a3-c9028d7792d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:34.888818 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.888728    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x78qb\" (UniqueName: \"kubernetes.io/projected/447fed40-9a8d-4a48-88a3-c9028d7792d5-kube-api-access-x78qb\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:34.888818 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.888761    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:34.888818 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.888772    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:34.888818 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:34.888780    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/447fed40-9a8d-4a48-88a3-c9028d7792d5-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:35.004127 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:35.004085    4553 generic.go:356] "Generic (PLEG): container finished" podID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerID="0be1c22599e8a6fb471786b544b570e4ab720231fe805d7f79a5ecd126ee4666" exitCode=0
Oct 30 22:11:35.004340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:35.004154    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"
Oct 30 22:11:35.004340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:35.004187    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s" event={"ID":"447fed40-9a8d-4a48-88a3-c9028d7792d5","Type":"ContainerDied","Data":"0be1c22599e8a6fb471786b544b570e4ab720231fe805d7f79a5ecd126ee4666"}
Oct 30 22:11:35.004340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:35.004250    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s" event={"ID":"447fed40-9a8d-4a48-88a3-c9028d7792d5","Type":"ContainerDied","Data":"48a29b60bcce943f184732509ed8cc4eb980084d74bc96b7ec8d7519acc4a174"}
Oct 30 22:11:35.004340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:35.004273    4553 scope.go:122] "RemoveContainer" containerID="0be1c22599e8a6fb471786b544b570e4ab720231fe805d7f79a5ecd126ee4666"
Oct 30 22:11:35.015288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:35.015268    4553 scope.go:122] "RemoveContainer" containerID="738e868126c7b7b0845d5bb9c136bf5a31aea1c488ec0b80a122019d1781bc21"
Oct 30 22:11:35.024525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:35.024506    4553 scope.go:122] "RemoveContainer" containerID="0be1c22599e8a6fb471786b544b570e4ab720231fe805d7f79a5ecd126ee4666"
Oct 30 22:11:35.024820 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:35.024799    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be1c22599e8a6fb471786b544b570e4ab720231fe805d7f79a5ecd126ee4666\": container with ID starting with 0be1c22599e8a6fb471786b544b570e4ab720231fe805d7f79a5ecd126ee4666 not found: ID does not exist" containerID="0be1c22599e8a6fb471786b544b570e4ab720231fe805d7f79a5ecd126ee4666"
Oct 30 22:11:35.024867 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:35.024831    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be1c22599e8a6fb471786b544b570e4ab720231fe805d7f79a5ecd126ee4666"} err="failed to get container status \"0be1c22599e8a6fb471786b544b570e4ab720231fe805d7f79a5ecd126ee4666\": rpc error: code = NotFound desc = could not find container \"0be1c22599e8a6fb471786b544b570e4ab720231fe805d7f79a5ecd126ee4666\": container with ID starting with 0be1c22599e8a6fb471786b544b570e4ab720231fe805d7f79a5ecd126ee4666 not found: ID does not exist"
Oct 30 22:11:35.024867 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:35.024850    4553 scope.go:122] "RemoveContainer" containerID="738e868126c7b7b0845d5bb9c136bf5a31aea1c488ec0b80a122019d1781bc21"
Oct 30 22:11:35.025085 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:35.025067    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738e868126c7b7b0845d5bb9c136bf5a31aea1c488ec0b80a122019d1781bc21\": container with ID starting with 738e868126c7b7b0845d5bb9c136bf5a31aea1c488ec0b80a122019d1781bc21 not found: ID does not exist" containerID="738e868126c7b7b0845d5bb9c136bf5a31aea1c488ec0b80a122019d1781bc21"
Oct 30 22:11:35.025177 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:35.025094    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738e868126c7b7b0845d5bb9c136bf5a31aea1c488ec0b80a122019d1781bc21"} err="failed to get container status \"738e868126c7b7b0845d5bb9c136bf5a31aea1c488ec0b80a122019d1781bc21\": rpc error: code = NotFound desc = could not find container \"738e868126c7b7b0845d5bb9c136bf5a31aea1c488ec0b80a122019d1781bc21\": container with ID starting with 738e868126c7b7b0845d5bb9c136bf5a31aea1c488ec0b80a122019d1781bc21 not found: ID does not exist"
Oct 30 22:11:35.176513 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:35.176412    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"]
Oct 30 22:11:35.180776 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:35.180746    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9ac4a7-openshift-default-5b885745c5-hjz5s"]
Oct 30 22:11:35.395084 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:35.395049    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="447fed40-9a8d-4a48-88a3-c9028d7792d5" path="/var/lib/kubelet/pods/447fed40-9a8d-4a48-88a3-c9028d7792d5/volumes"
Oct 30 22:11:36.569849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:36.569814    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"]
Oct 30 22:11:36.570400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:36.570084    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4" podUID="ee288017-2408-4f01-bcff-1a575529a750" containerName="istio-proxy" containerID="cri-o://490247fbebf06adffec50addafeaf0cbf6f0c0010792b0d2d4394d87554b1b7f" gracePeriod=30
Oct 30 22:11:41.852436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.852355    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:11:41.885828 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.885792    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee288017-2408-4f01-bcff-1a575529a750-istio-token\" (UniqueName: \"kubernetes.io/projected/ee288017-2408-4f01-bcff-1a575529a750-istio-token\") pod \"ee288017-2408-4f01-bcff-1a575529a750\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") "
Oct 30 22:11:41.886040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.885912    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee288017-2408-4f01-bcff-1a575529a750-kube-api-access-96xqg\" (UniqueName: \"kubernetes.io/projected/ee288017-2408-4f01-bcff-1a575529a750-kube-api-access-96xqg\") pod \"ee288017-2408-4f01-bcff-1a575529a750\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") "
Oct 30 22:11:41.886040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.885942    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee288017-2408-4f01-bcff-1a575529a750-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee288017-2408-4f01-bcff-1a575529a750-istiod-ca-cert\") pod \"ee288017-2408-4f01-bcff-1a575529a750\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") "
Oct 30 22:11:41.886040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.885975    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-workload-certs\") pod \"ee288017-2408-4f01-bcff-1a575529a750\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") "
Oct 30 22:11:41.886040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.886000    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-workload-socket\") pod \"ee288017-2408-4f01-bcff-1a575529a750\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") "
Oct 30 22:11:41.886286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.886048    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-istio-envoy\") pod \"ee288017-2408-4f01-bcff-1a575529a750\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") "
Oct 30 22:11:41.886286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.886241    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-istio-data\") pod \"ee288017-2408-4f01-bcff-1a575529a750\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") "
Oct 30 22:11:41.886389 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.886300    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee288017-2408-4f01-bcff-1a575529a750-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee288017-2408-4f01-bcff-1a575529a750-istio-podinfo\") pod \"ee288017-2408-4f01-bcff-1a575529a750\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") "
Oct 30 22:11:41.886389 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.886315    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-workload-certs" pod "ee288017-2408-4f01-bcff-1a575529a750" (UID: "ee288017-2408-4f01-bcff-1a575529a750"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:41.886389 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.886344    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-credential-socket\") pod \"ee288017-2408-4f01-bcff-1a575529a750\" (UID: \"ee288017-2408-4f01-bcff-1a575529a750\") "
Oct 30 22:11:41.886535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.886456    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee288017-2408-4f01-bcff-1a575529a750-istiod-ca-cert" pod "ee288017-2408-4f01-bcff-1a575529a750" (UID: "ee288017-2408-4f01-bcff-1a575529a750"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:41.886535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.886438    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-workload-socket" pod "ee288017-2408-4f01-bcff-1a575529a750" (UID: "ee288017-2408-4f01-bcff-1a575529a750"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:41.886535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.886491    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-istio-data" pod "ee288017-2408-4f01-bcff-1a575529a750" (UID: "ee288017-2408-4f01-bcff-1a575529a750"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:41.886763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.886726    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-credential-socket" pod "ee288017-2408-4f01-bcff-1a575529a750" (UID: "ee288017-2408-4f01-bcff-1a575529a750"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:41.887205 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.887121    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:41.887319 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.887211    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:41.887319 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.887228    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee288017-2408-4f01-bcff-1a575529a750-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:41.887319 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.887244    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:41.887319 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.887258    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:41.888485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.888458    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-istio-envoy" pod "ee288017-2408-4f01-bcff-1a575529a750" (UID: "ee288017-2408-4f01-bcff-1a575529a750"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:41.888598 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.888504    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee288017-2408-4f01-bcff-1a575529a750-istio-podinfo" pod "ee288017-2408-4f01-bcff-1a575529a750" (UID: "ee288017-2408-4f01-bcff-1a575529a750"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:41.888598 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.888534    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee288017-2408-4f01-bcff-1a575529a750-istio-token" pod "ee288017-2408-4f01-bcff-1a575529a750" (UID: "ee288017-2408-4f01-bcff-1a575529a750"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:41.888986 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.888968    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee288017-2408-4f01-bcff-1a575529a750-kube-api-access-96xqg" pod "ee288017-2408-4f01-bcff-1a575529a750" (UID: "ee288017-2408-4f01-bcff-1a575529a750"). InnerVolumeSpecName "kube-api-access-96xqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:41.988407 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.988365    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee288017-2408-4f01-bcff-1a575529a750-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:41.988407 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.988398    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee288017-2408-4f01-bcff-1a575529a750-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:41.988407 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.988412    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96xqg\" (UniqueName: \"kubernetes.io/projected/ee288017-2408-4f01-bcff-1a575529a750-kube-api-access-96xqg\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:41.988667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:41.988426    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee288017-2408-4f01-bcff-1a575529a750-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:42.054510 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:42.054472    4553 generic.go:356] "Generic (PLEG): container finished" podID="ee288017-2408-4f01-bcff-1a575529a750" containerID="490247fbebf06adffec50addafeaf0cbf6f0c0010792b0d2d4394d87554b1b7f" exitCode=0
Oct 30 22:11:42.054699 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:42.054669    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4" event={"ID":"ee288017-2408-4f01-bcff-1a575529a750","Type":"ContainerDied","Data":"490247fbebf06adffec50addafeaf0cbf6f0c0010792b0d2d4394d87554b1b7f"}
Oct 30 22:11:42.054789 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:42.054712    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4" event={"ID":"ee288017-2408-4f01-bcff-1a575529a750","Type":"ContainerDied","Data":"6ecf1787b92ae9948623e72741310a67145c4aa1101e7de6120faf4825d24972"}
Oct 30 22:11:42.054789 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:42.054738    4553 scope.go:122] "RemoveContainer" containerID="490247fbebf06adffec50addafeaf0cbf6f0c0010792b0d2d4394d87554b1b7f"
Oct 30 22:11:42.054893 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:42.054880    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"
Oct 30 22:11:42.078840 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:42.078801    4553 scope.go:122] "RemoveContainer" containerID="e7b06d9e6b79b59e93944855db84920ce7d761854c4fddc7e606f1143ec14b56"
Oct 30 22:11:42.088740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:42.088717    4553 scope.go:122] "RemoveContainer" containerID="490247fbebf06adffec50addafeaf0cbf6f0c0010792b0d2d4394d87554b1b7f"
Oct 30 22:11:42.089071 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:42.089044    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490247fbebf06adffec50addafeaf0cbf6f0c0010792b0d2d4394d87554b1b7f\": container with ID starting with 490247fbebf06adffec50addafeaf0cbf6f0c0010792b0d2d4394d87554b1b7f not found: ID does not exist" containerID="490247fbebf06adffec50addafeaf0cbf6f0c0010792b0d2d4394d87554b1b7f"
Oct 30 22:11:42.089197 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:42.089079    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490247fbebf06adffec50addafeaf0cbf6f0c0010792b0d2d4394d87554b1b7f"} err="failed to get container status \"490247fbebf06adffec50addafeaf0cbf6f0c0010792b0d2d4394d87554b1b7f\": rpc error: code = NotFound desc = could not find container \"490247fbebf06adffec50addafeaf0cbf6f0c0010792b0d2d4394d87554b1b7f\": container with ID starting with 490247fbebf06adffec50addafeaf0cbf6f0c0010792b0d2d4394d87554b1b7f not found: ID does not exist"
Oct 30 22:11:42.089197 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:42.089098    4553 scope.go:122] "RemoveContainer" containerID="e7b06d9e6b79b59e93944855db84920ce7d761854c4fddc7e606f1143ec14b56"
Oct 30 22:11:42.089447 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:42.089408    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b06d9e6b79b59e93944855db84920ce7d761854c4fddc7e606f1143ec14b56\": container with ID starting with e7b06d9e6b79b59e93944855db84920ce7d761854c4fddc7e606f1143ec14b56 not found: ID does not exist" containerID="e7b06d9e6b79b59e93944855db84920ce7d761854c4fddc7e606f1143ec14b56"
Oct 30 22:11:42.089447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:42.089435    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b06d9e6b79b59e93944855db84920ce7d761854c4fddc7e606f1143ec14b56"} err="failed to get container status \"e7b06d9e6b79b59e93944855db84920ce7d761854c4fddc7e606f1143ec14b56\": rpc error: code = NotFound desc = could not find container \"e7b06d9e6b79b59e93944855db84920ce7d761854c4fddc7e606f1143ec14b56\": container with ID starting with e7b06d9e6b79b59e93944855db84920ce7d761854c4fddc7e606f1143ec14b56 not found: ID does not exist"
Oct 30 22:11:42.244500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:42.244403    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"]
Oct 30 22:11:42.254811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:42.254771    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-da0f3d-openshift-default-d6fdb46d9-lnxk4"]
Oct 30 22:11:43.142752 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:43.142717    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"]
Oct 30 22:11:43.393791 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:43.393706    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee288017-2408-4f01-bcff-1a575529a750" path="/var/lib/kubelet/pods/ee288017-2408-4f01-bcff-1a575529a750/volumes"
Oct 30 22:11:48.144007 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:48.143962    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8" podUID="e5b5b84d-ccf6-4683-af90-ba5aef6c821e" containerName="payload-processing" containerID="cri-o://7b4a88ba55a0c0ee2c46e6decae26e5fe821354cabb78d460b97e1bbdf835fad" gracePeriod=25
Oct 30 22:11:48.413825 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:48.413746    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"
Oct 30 22:11:48.473479 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:48.473113    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b5b84d-ccf6-4683-af90-ba5aef6c821e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b5b84d-ccf6-4683-af90-ba5aef6c821e-plugins-config-volume\") pod \"e5b5b84d-ccf6-4683-af90-ba5aef6c821e\" (UID: \"e5b5b84d-ccf6-4683-af90-ba5aef6c821e\") "
Oct 30 22:11:48.473479 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:48.473263    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b5b84d-ccf6-4683-af90-ba5aef6c821e-kube-api-access-855wx\" (UniqueName: \"kubernetes.io/projected/e5b5b84d-ccf6-4683-af90-ba5aef6c821e-kube-api-access-855wx\") pod \"e5b5b84d-ccf6-4683-af90-ba5aef6c821e\" (UID: \"e5b5b84d-ccf6-4683-af90-ba5aef6c821e\") "
Oct 30 22:11:48.474439 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:48.474412    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b5b84d-ccf6-4683-af90-ba5aef6c821e-plugins-config-volume" pod "e5b5b84d-ccf6-4683-af90-ba5aef6c821e" (UID: "e5b5b84d-ccf6-4683-af90-ba5aef6c821e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:48.476435 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:48.476409    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b5b84d-ccf6-4683-af90-ba5aef6c821e-kube-api-access-855wx" pod "e5b5b84d-ccf6-4683-af90-ba5aef6c821e" (UID: "e5b5b84d-ccf6-4683-af90-ba5aef6c821e"). InnerVolumeSpecName "kube-api-access-855wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:48.576179 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:48.576117    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b5b84d-ccf6-4683-af90-ba5aef6c821e-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:48.576179 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:48.576181    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-855wx\" (UniqueName: \"kubernetes.io/projected/e5b5b84d-ccf6-4683-af90-ba5aef6c821e-kube-api-access-855wx\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:49.100706 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:49.100670    4553 generic.go:356] "Generic (PLEG): container finished" podID="e5b5b84d-ccf6-4683-af90-ba5aef6c821e" containerID="7b4a88ba55a0c0ee2c46e6decae26e5fe821354cabb78d460b97e1bbdf835fad" exitCode=0
Oct 30 22:11:49.100872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:49.100734    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"
Oct 30 22:11:49.100872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:49.100753    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8" event={"ID":"e5b5b84d-ccf6-4683-af90-ba5aef6c821e","Type":"ContainerDied","Data":"7b4a88ba55a0c0ee2c46e6decae26e5fe821354cabb78d460b97e1bbdf835fad"}
Oct 30 22:11:49.100872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:49.100811    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8" event={"ID":"e5b5b84d-ccf6-4683-af90-ba5aef6c821e","Type":"ContainerDied","Data":"891695c653c3cbfa42c935d365b94e4268d5d108e4cc49b174e50e39d233de67"}
Oct 30 22:11:49.100872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:49.100839    4553 scope.go:122] "RemoveContainer" containerID="7b4a88ba55a0c0ee2c46e6decae26e5fe821354cabb78d460b97e1bbdf835fad"
Oct 30 22:11:49.111585 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:49.111564    4553 scope.go:122] "RemoveContainer" containerID="7b4a88ba55a0c0ee2c46e6decae26e5fe821354cabb78d460b97e1bbdf835fad"
Oct 30 22:11:49.111912 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:49.111893    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4a88ba55a0c0ee2c46e6decae26e5fe821354cabb78d460b97e1bbdf835fad\": container with ID starting with 7b4a88ba55a0c0ee2c46e6decae26e5fe821354cabb78d460b97e1bbdf835fad not found: ID does not exist" containerID="7b4a88ba55a0c0ee2c46e6decae26e5fe821354cabb78d460b97e1bbdf835fad"
Oct 30 22:11:49.112004 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:49.111923    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4a88ba55a0c0ee2c46e6decae26e5fe821354cabb78d460b97e1bbdf835fad"} err="failed to get container status \"7b4a88ba55a0c0ee2c46e6decae26e5fe821354cabb78d460b97e1bbdf835fad\": rpc error: code = NotFound desc = could not find container \"7b4a88ba55a0c0ee2c46e6decae26e5fe821354cabb78d460b97e1bbdf835fad\": container with ID starting with 7b4a88ba55a0c0ee2c46e6decae26e5fe821354cabb78d460b97e1bbdf835fad not found: ID does not exist"
Oct 30 22:11:49.301893 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:49.301861    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"]
Oct 30 22:11:49.352525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:49.352435    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-02a8fa-6987989665-drfl8"]
Oct 30 22:11:49.393483 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:49.393437    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b5b84d-ccf6-4683-af90-ba5aef6c821e" path="/var/lib/kubelet/pods/e5b5b84d-ccf6-4683-af90-ba5aef6c821e/volumes"
Oct 30 22:11:50.548733 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.548699    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"]
Oct 30 22:11:50.549280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549266    4553 cpu_manager.go:414] "containerMap: removing container" podUID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" containerName="istio-proxy"
Oct 30 22:11:50.549371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549281    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" containerName="istio-proxy"
Oct 30 22:11:50.549371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549296    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy"
Oct 30 22:11:50.549371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549301    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy"
Oct 30 22:11:50.549371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549313    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9e4d1744-6b44-46ae-8e36-0f3584c10ed6" containerName="maas-api"
Oct 30 22:11:50.549371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549317    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e4d1744-6b44-46ae-8e36-0f3584c10ed6" containerName="maas-api"
Oct 30 22:11:50.549371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549328    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ee288017-2408-4f01-bcff-1a575529a750" containerName="istio-proxy"
Oct 30 22:11:50.549371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549334    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee288017-2408-4f01-bcff-1a575529a750" containerName="istio-proxy"
Oct 30 22:11:50.549371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549341    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ee288017-2408-4f01-bcff-1a575529a750" containerName="istio-proxy"
Oct 30 22:11:50.549371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549345    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee288017-2408-4f01-bcff-1a575529a750" containerName="istio-proxy"
Oct 30 22:11:50.549371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549354    4553 cpu_manager.go:414] "containerMap: removing container" podUID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerName="istio-proxy"
Oct 30 22:11:50.549371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549358    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerName="istio-proxy"
Oct 30 22:11:50.549371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549365    4553 cpu_manager.go:414] "containerMap: removing container" podUID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" containerName="istio-proxy"
Oct 30 22:11:50.549371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549369    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549381    4553 cpu_manager.go:414] "containerMap: removing container" podUID="fa482183-5d9f-461a-af5d-b30fd2ed8428" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549385    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa482183-5d9f-461a-af5d-b30fd2ed8428" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549395    4553 cpu_manager.go:414] "containerMap: removing container" podUID="8324c7c5-ac7a-439d-9f12-048a609e6ab3" containerName="maas-api"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549399    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8324c7c5-ac7a-439d-9f12-048a609e6ab3" containerName="maas-api"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549407    4553 cpu_manager.go:414] "containerMap: removing container" podUID="3e4c8521-f6ec-4d79-8946-045048bdad39" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549411    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e4c8521-f6ec-4d79-8946-045048bdad39" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549418    4553 cpu_manager.go:414] "containerMap: removing container" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549422    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549428    4553 cpu_manager.go:414] "containerMap: removing container" podUID="20775493-c504-4316-9d6f-816952c83e9a" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549432    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20775493-c504-4316-9d6f-816952c83e9a" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549439    4553 cpu_manager.go:414] "containerMap: removing container" podUID="4bb761f4-75aa-481c-ad0a-0cfc782028ac" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549444    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb761f4-75aa-481c-ad0a-0cfc782028ac" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549452    4553 cpu_manager.go:414] "containerMap: removing container" podUID="3267e456-b8e8-43be-9360-f75afeaadb7b" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549456    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3267e456-b8e8-43be-9360-f75afeaadb7b" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549463    4553 cpu_manager.go:414] "containerMap: removing container" podUID="c2b6e606-c225-4dd2-94c0-e8bfe42ee109" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549467    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b6e606-c225-4dd2-94c0-e8bfe42ee109" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549473    4553 cpu_manager.go:414] "containerMap: removing container" podUID="767d4fe8-a9f3-4627-bd29-68732f291535" containerName="payload-processing"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549477    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767d4fe8-a9f3-4627-bd29-68732f291535" containerName="payload-processing"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549487    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9aa7dcc5-d42a-4f74-8452-542b6ed5b404" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549491    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aa7dcc5-d42a-4f74-8452-542b6ed5b404" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549498    4553 cpu_manager.go:414] "containerMap: removing container" podUID="e5b5b84d-ccf6-4683-af90-ba5aef6c821e" containerName="payload-processing"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549502    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b5b84d-ccf6-4683-af90-ba5aef6c821e" containerName="payload-processing"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549511    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7ee10de5-d1f3-47dd-a0e8-a97e11381b1e" containerName="payload-pre-processing"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549515    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee10de5-d1f3-47dd-a0e8-a97e11381b1e" containerName="payload-pre-processing"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549595    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549605    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549612    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa482183-5d9f-461a-af5d-b30fd2ed8428" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549619    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="8324c7c5-ac7a-439d-9f12-048a609e6ab3" containerName="maas-api"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549625    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e4c8521-f6ec-4d79-8946-045048bdad39" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549631    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc77e33-55c0-4f13-8ff1-3ab8706e4ec7" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549649    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="20775493-c504-4316-9d6f-816952c83e9a" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549654    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb761f4-75aa-481c-ad0a-0cfc782028ac" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549660    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="3267e456-b8e8-43be-9360-f75afeaadb7b" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549666    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b6e606-c225-4dd2-94c0-e8bfe42ee109" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549671    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="767d4fe8-a9f3-4627-bd29-68732f291535" containerName="payload-processing"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549676    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aa7dcc5-d42a-4f74-8452-542b6ed5b404" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549682    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b5b84d-ccf6-4683-af90-ba5aef6c821e" containerName="payload-processing"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549687    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee10de5-d1f3-47dd-a0e8-a97e11381b1e" containerName="payload-pre-processing"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549694    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d66e97b1-d8b1-477a-a8f7-82466dbcd432" containerName="istio-proxy"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549700    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e4d1744-6b44-46ae-8e36-0f3584c10ed6" containerName="maas-api"
Oct 30 22:11:50.549744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.549705    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee288017-2408-4f01-bcff-1a575529a750" containerName="istio-proxy"
Oct 30 22:11:50.553254 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.553232    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"
Oct 30 22:11:50.619373 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.619337    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"]
Oct 30 22:11:50.620249 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.620223    4553 cpu_manager.go:414] "containerMap: removing container" podUID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerName="istio-proxy"
Oct 30 22:11:50.620249 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.620246    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerName="istio-proxy"
Oct 30 22:11:50.620423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.620339    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee288017-2408-4f01-bcff-1a575529a750" containerName="istio-proxy"
Oct 30 22:11:50.620423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.620347    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="447fed40-9a8d-4a48-88a3-c9028d7792d5" containerName="istio-proxy"
Oct 30 22:11:50.620423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.620358    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="2747809c-ab61-4e18-8cd3-e6bcfe0de5c3" containerName="istio-proxy"
Oct 30 22:11:50.623794 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.623771    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"
Oct 30 22:11:50.665245 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.665193    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7fabe91d-dockercfg-bqrsq\""
Oct 30 22:11:50.665245 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.665247    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7fabe91d\""
Oct 30 22:11:50.720660 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.720614    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcfz8\" (UniqueName: \"kubernetes.io/projected/e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c-kube-api-access-zcfz8\") pod \"payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4\" (UID: \"e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c\") " pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"
Oct 30 22:11:50.724578 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.724530    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c-plugins-config-volume\") pod \"payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4\" (UID: \"e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c\") " pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"
Oct 30 22:11:50.724781 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.724697    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lhbq\" (UniqueName: \"kubernetes.io/projected/13c0c4e6-7049-4860-a606-4fb171299052-kube-api-access-7lhbq\") pod \"payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6\" (UID: \"13c0c4e6-7049-4860-a606-4fb171299052\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"
Oct 30 22:11:50.741160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.724894    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c0c4e6-7049-4860-a606-4fb171299052-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6\" (UID: \"13c0c4e6-7049-4860-a606-4fb171299052\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"
Oct 30 22:11:50.843246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.830520    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c0c4e6-7049-4860-a606-4fb171299052-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6\" (UID: \"13c0c4e6-7049-4860-a606-4fb171299052\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"
Oct 30 22:11:50.843246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.830650    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcfz8\" (UniqueName: \"kubernetes.io/projected/e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c-kube-api-access-zcfz8\") pod \"payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4\" (UID: \"e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c\") " pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"
Oct 30 22:11:50.843246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.830725    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c-plugins-config-volume\") pod \"payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4\" (UID: \"e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c\") " pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"
Oct 30 22:11:50.843246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.830810    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lhbq\" (UniqueName: \"kubernetes.io/projected/13c0c4e6-7049-4860-a606-4fb171299052-kube-api-access-7lhbq\") pod \"payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6\" (UID: \"13c0c4e6-7049-4860-a606-4fb171299052\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"
Oct 30 22:11:50.843246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.832585    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c0c4e6-7049-4860-a606-4fb171299052-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6\" (UID: \"13c0c4e6-7049-4860-a606-4fb171299052\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"
Oct 30 22:11:50.843246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.833599    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c-plugins-config-volume\") pod \"payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4\" (UID: \"e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c\") " pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"
Oct 30 22:11:50.846297 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.846245    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"]
Oct 30 22:11:50.850400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.850376    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"]
Oct 30 22:11:50.893803 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.893766    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcfz8\" (UniqueName: \"kubernetes.io/projected/e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c-kube-api-access-zcfz8\") pod \"payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4\" (UID: \"e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c\") " pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"
Oct 30 22:11:50.893946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.893889    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lhbq\" (UniqueName: \"kubernetes.io/projected/13c0c4e6-7049-4860-a606-4fb171299052-kube-api-access-7lhbq\") pod \"payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6\" (UID: \"13c0c4e6-7049-4860-a606-4fb171299052\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"
Oct 30 22:11:50.940476 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:50.940448    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"
Oct 30 22:11:51.169017 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:51.168920    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"
Oct 30 22:11:51.329635 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:51.329606    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"]
Oct 30 22:11:51.330463 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:11:51.330436    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode994e348_81a2_4de2_9aa3_ca9fe3dc0a9c.slice/crio-5002e5508ecf3a09a7936dca0878d193b3fff7539198c88c889712116819c85d WatchSource:0}: Error finding container 5002e5508ecf3a09a7936dca0878d193b3fff7539198c88c889712116819c85d: Status 404 returned error can't find the container with id 5002e5508ecf3a09a7936dca0878d193b3fff7539198c88c889712116819c85d
Oct 30 22:11:51.580821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:51.580792    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"]
Oct 30 22:11:51.581685 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:11:51.581662    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c0c4e6_7049_4860_a606_4fb171299052.slice/crio-1535b4f4de169719d9949b5a667490a74f19d25c29058f20c1acaf3f73db8a03 WatchSource:0}: Error finding container 1535b4f4de169719d9949b5a667490a74f19d25c29058f20c1acaf3f73db8a03: Status 404 returned error can't find the container with id 1535b4f4de169719d9949b5a667490a74f19d25c29058f20c1acaf3f73db8a03
Oct 30 22:11:52.121540 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:52.121502    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4" event={"ID":"e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c","Type":"ContainerStarted","Data":"50c6189376b3c21b7a316d98a96914aee9f31b08be70e8db5f583d6f5fe76141"}
Oct 30 22:11:52.121763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:52.121573    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"
Oct 30 22:11:52.121763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:52.121598    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4" event={"ID":"e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c","Type":"ContainerStarted","Data":"5002e5508ecf3a09a7936dca0878d193b3fff7539198c88c889712116819c85d"}
Oct 30 22:11:52.123445 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:52.123416    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6" event={"ID":"13c0c4e6-7049-4860-a606-4fb171299052","Type":"ContainerStarted","Data":"e9eb8a6212d4dd1f66885f964c19ac3f943d689bf91fa9a20de9ae211f9cd8df"}
Oct 30 22:11:52.123585 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:52.123465    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6" event={"ID":"13c0c4e6-7049-4860-a606-4fb171299052","Type":"ContainerStarted","Data":"1535b4f4de169719d9949b5a667490a74f19d25c29058f20c1acaf3f73db8a03"}
Oct 30 22:11:52.123585 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:52.123492    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"
Oct 30 22:11:52.249592 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:52.249538    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4" podStartSLOduration=2.249523346 podStartE2EDuration="2.249523346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:52.245938884 +0000 UTC m=+1762.980301915" watchObservedRunningTime="2026-10-30 22:11:52.249523346 +0000 UTC m=+1762.983886376"
Oct 30 22:11:53.876719 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:53.876656    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6" podStartSLOduration=3.876639029 podStartE2EDuration="3.876639029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:52.370957568 +0000 UTC m=+1763.105320596" watchObservedRunningTime="2026-10-30 22:11:53.876639029 +0000 UTC m=+1764.611002059"
Oct 30 22:11:53.879520 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:53.879495    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"]
Oct 30 22:11:54.031636 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:54.031604    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"]
Oct 30 22:11:56.355920 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:56.355885    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"]
Oct 30 22:11:56.360337 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:56.360314    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"
Oct 30 22:11:56.419682 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:56.419648    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"]
Oct 30 22:11:56.435365 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:56.435321    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41752921-dba6-4eec-ac05-1971cba00908-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692\" (UID: \"41752921-dba6-4eec-ac05-1971cba00908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"
Oct 30 22:11:56.435553 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:56.435515    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvw95\" (UniqueName: \"kubernetes.io/projected/41752921-dba6-4eec-ac05-1971cba00908-kube-api-access-vvw95\") pod \"payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692\" (UID: \"41752921-dba6-4eec-ac05-1971cba00908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"
Oct 30 22:11:56.465431 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:56.465400    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-97ca39-dockercfg-rm28h\""
Oct 30 22:11:56.465431 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:56.465429    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-97ca39\""
Oct 30 22:11:56.537218 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:56.537181    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvw95\" (UniqueName: \"kubernetes.io/projected/41752921-dba6-4eec-ac05-1971cba00908-kube-api-access-vvw95\") pod \"payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692\" (UID: \"41752921-dba6-4eec-ac05-1971cba00908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"
Oct 30 22:11:56.537503 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:56.537480    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41752921-dba6-4eec-ac05-1971cba00908-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692\" (UID: \"41752921-dba6-4eec-ac05-1971cba00908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"
Oct 30 22:11:56.538262 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:56.538239    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41752921-dba6-4eec-ac05-1971cba00908-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692\" (UID: \"41752921-dba6-4eec-ac05-1971cba00908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"
Oct 30 22:11:56.603396 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:56.603360    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvw95\" (UniqueName: \"kubernetes.io/projected/41752921-dba6-4eec-ac05-1971cba00908-kube-api-access-vvw95\") pod \"payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692\" (UID: \"41752921-dba6-4eec-ac05-1971cba00908\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"
Oct 30 22:11:56.608459 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:56.608389    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"]
Oct 30 22:11:56.674760 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:56.674729    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"
Oct 30 22:11:57.088500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:57.085121    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"]
Oct 30 22:11:57.111928 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:57.111895    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"]
Oct 30 22:11:57.117552 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:57.117528    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"
Oct 30 22:11:57.155026 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:57.154992    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2c8fd1-3f67-423d-92a7-40923ce4848a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww\" (UID: \"5b2c8fd1-3f67-423d-92a7-40923ce4848a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"
Oct 30 22:11:57.155257 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:57.155210    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4sjm\" (UniqueName: \"kubernetes.io/projected/5b2c8fd1-3f67-423d-92a7-40923ce4848a-kube-api-access-l4sjm\") pod \"payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww\" (UID: \"5b2c8fd1-3f67-423d-92a7-40923ce4848a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"
Oct 30 22:11:57.155339 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:57.155286    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692" event={"ID":"41752921-dba6-4eec-ac05-1971cba00908","Type":"ContainerStarted","Data":"115e9f1343a16a24fe4a459bbc645df7d1ada7db63d41703e61d6124d713e170"}
Oct 30 22:11:57.177208 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:57.177125    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"]
Oct 30 22:11:57.257211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:57.257156    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4sjm\" (UniqueName: \"kubernetes.io/projected/5b2c8fd1-3f67-423d-92a7-40923ce4848a-kube-api-access-l4sjm\") pod \"payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww\" (UID: \"5b2c8fd1-3f67-423d-92a7-40923ce4848a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"
Oct 30 22:11:57.257431 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:57.257407    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2c8fd1-3f67-423d-92a7-40923ce4848a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww\" (UID: \"5b2c8fd1-3f67-423d-92a7-40923ce4848a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"
Oct 30 22:11:57.258113 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:57.258088    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2c8fd1-3f67-423d-92a7-40923ce4848a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww\" (UID: \"5b2c8fd1-3f67-423d-92a7-40923ce4848a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"
Oct 30 22:11:57.315406 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:57.315375    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4sjm\" (UniqueName: \"kubernetes.io/projected/5b2c8fd1-3f67-423d-92a7-40923ce4848a-kube-api-access-l4sjm\") pod \"payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww\" (UID: \"5b2c8fd1-3f67-423d-92a7-40923ce4848a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"
Oct 30 22:11:57.432982 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:57.432876    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"
Oct 30 22:11:57.814395 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:57.814367    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"]
Oct 30 22:11:57.815731 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:11:57.815702    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2c8fd1_3f67_423d_92a7_40923ce4848a.slice/crio-57bcca9d7aff97f5a9d9c16fdc5b99ff45c0cf15e1a85616c9d85f04c1eb02a2 WatchSource:0}: Error finding container 57bcca9d7aff97f5a9d9c16fdc5b99ff45c0cf15e1a85616c9d85f04c1eb02a2: Status 404 returned error can't find the container with id 57bcca9d7aff97f5a9d9c16fdc5b99ff45c0cf15e1a85616c9d85f04c1eb02a2
Oct 30 22:11:58.180532 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:58.180490    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww" event={"ID":"5b2c8fd1-3f67-423d-92a7-40923ce4848a","Type":"ContainerStarted","Data":"57bcca9d7aff97f5a9d9c16fdc5b99ff45c0cf15e1a85616c9d85f04c1eb02a2"}
Oct 30 22:11:58.218285 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:11:58.218224    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:11:58.879877 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:58.879805    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp" podUID="f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd" containerName="payload-processing" containerID="cri-o://0773f0d07c503dc884cbb01ecadd309ebc074d6bd503a36b40fef3910983d8da" gracePeriod=25
Oct 30 22:11:59.033336 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.033271    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx" podUID="cc64ce9a-ba29-44aa-a770-f98d29183190" containerName="payload-pre-processing" containerID="cri-o://89664c8b57a0ef4ecebcf59deddd357cd8fa41092ee0c90b01737d2bb7260014" gracePeriod=25
Oct 30 22:11:59.195518 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.195365    4553 generic.go:356] "Generic (PLEG): container finished" podID="f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd" containerID="0773f0d07c503dc884cbb01ecadd309ebc074d6bd503a36b40fef3910983d8da" exitCode=0
Oct 30 22:11:59.195518 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.195468    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp" event={"ID":"f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd","Type":"ContainerDied","Data":"0773f0d07c503dc884cbb01ecadd309ebc074d6bd503a36b40fef3910983d8da"}
Oct 30 22:11:59.195766 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.195615    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp" event={"ID":"f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd","Type":"ContainerDied","Data":"0cb3117e2c0ef43dc2e693539f5b87b76ed59ff1bcab4bfd9d453cc693ca757e"}
Oct 30 22:11:59.195766 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.195664    4553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cb3117e2c0ef43dc2e693539f5b87b76ed59ff1bcab4bfd9d453cc693ca757e"
Oct 30 22:11:59.196397 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.196377    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"
Oct 30 22:11:59.199513 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.199298    4553 generic.go:356] "Generic (PLEG): container finished" podID="cc64ce9a-ba29-44aa-a770-f98d29183190" containerID="89664c8b57a0ef4ecebcf59deddd357cd8fa41092ee0c90b01737d2bb7260014" exitCode=0
Oct 30 22:11:59.199513 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.199459    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx" event={"ID":"cc64ce9a-ba29-44aa-a770-f98d29183190","Type":"ContainerDied","Data":"89664c8b57a0ef4ecebcf59deddd357cd8fa41092ee0c90b01737d2bb7260014"}
Oct 30 22:11:59.297153 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.297102    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd-kube-api-access-rpmnh\" (UniqueName: \"kubernetes.io/projected/f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd-kube-api-access-rpmnh\") pod \"f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd\" (UID: \"f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd\") "
Oct 30 22:11:59.297461 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.297408    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd-plugins-config-volume\") pod \"f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd\" (UID: \"f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd\") "
Oct 30 22:11:59.298752 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.298719    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd-plugins-config-volume" pod "f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd" (UID: "f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:59.319030 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.317774    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd-kube-api-access-rpmnh" pod "f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd" (UID: "f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd"). InnerVolumeSpecName "kube-api-access-rpmnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:59.364821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.364784    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"
Oct 30 22:11:59.399038 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.398985    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpmnh\" (UniqueName: \"kubernetes.io/projected/f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd-kube-api-access-rpmnh\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:59.399038 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.399013    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:59.500432 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.500400    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc64ce9a-ba29-44aa-a770-f98d29183190-kube-api-access-mfdgn\" (UniqueName: \"kubernetes.io/projected/cc64ce9a-ba29-44aa-a770-f98d29183190-kube-api-access-mfdgn\") pod \"cc64ce9a-ba29-44aa-a770-f98d29183190\" (UID: \"cc64ce9a-ba29-44aa-a770-f98d29183190\") "
Oct 30 22:11:59.500616 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.500460    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc64ce9a-ba29-44aa-a770-f98d29183190-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc64ce9a-ba29-44aa-a770-f98d29183190-plugins-config-volume\") pod \"cc64ce9a-ba29-44aa-a770-f98d29183190\" (UID: \"cc64ce9a-ba29-44aa-a770-f98d29183190\") "
Oct 30 22:11:59.501269 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.501233    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc64ce9a-ba29-44aa-a770-f98d29183190-plugins-config-volume" pod "cc64ce9a-ba29-44aa-a770-f98d29183190" (UID: "cc64ce9a-ba29-44aa-a770-f98d29183190"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:59.504025 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.503995    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc64ce9a-ba29-44aa-a770-f98d29183190-kube-api-access-mfdgn" pod "cc64ce9a-ba29-44aa-a770-f98d29183190" (UID: "cc64ce9a-ba29-44aa-a770-f98d29183190"). InnerVolumeSpecName "kube-api-access-mfdgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:59.602600 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.602561    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfdgn\" (UniqueName: \"kubernetes.io/projected/cc64ce9a-ba29-44aa-a770-f98d29183190-kube-api-access-mfdgn\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:59.602600 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:11:59.602593    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc64ce9a-ba29-44aa-a770-f98d29183190-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:00.209118 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:00.209086    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"
Oct 30 22:12:00.209593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:00.209108    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx" event={"ID":"cc64ce9a-ba29-44aa-a770-f98d29183190","Type":"ContainerDied","Data":"7c52842ac67e11e90b807548cb8922f27cf37080183ac02c48c2ae8e29fa5611"}
Oct 30 22:12:00.209593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:00.209229    4553 scope.go:122] "RemoveContainer" containerID="89664c8b57a0ef4ecebcf59deddd357cd8fa41092ee0c90b01737d2bb7260014"
Oct 30 22:12:00.211190 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:00.211124    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692" event={"ID":"41752921-dba6-4eec-ac05-1971cba00908","Type":"ContainerStarted","Data":"fca8dc48e6576317abd3611f850a669608ec3c59a52e0cbcfb68050c9bcb1152"}
Oct 30 22:12:00.211326 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:00.211215    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"
Oct 30 22:12:00.211326 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:00.211238    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"
Oct 30 22:12:00.383815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:00.383779    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"]
Oct 30 22:12:00.391065 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:00.390982    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d73ad7-5ccfcbb867-g7mgx"]
Oct 30 22:12:00.559879 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:00.559823    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692" podStartSLOduration=1.778745735 podStartE2EDuration="4.559807756s" totalImagesPullingTime="2.781062021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:00.558989848 +0000 UTC m=+1771.293352880" watchObservedRunningTime="2026-10-30 22:12:00.559807756 +0000 UTC m=+1771.294170786"
Oct 30 22:12:00.715221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:00.709207    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"]
Oct 30 22:12:00.731732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:00.728819    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d73ad7-84cf749d95-786wp"]
Oct 30 22:12:01.223273 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:01.223236    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww" event={"ID":"5b2c8fd1-3f67-423d-92a7-40923ce4848a","Type":"ContainerStarted","Data":"ce06f58525bc48c640a4a23ac7b97c088b2f98d3d95facc165ee66500306ded9"}
Oct 30 22:12:01.223765 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:01.223535    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"
Oct 30 22:12:01.353467 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:01.353409    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww" podStartSLOduration=2.032241809 podStartE2EDuration="4.353391144s" totalImagesPullingTime="2.321149335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:01.351491236 +0000 UTC m=+1772.085854288" watchObservedRunningTime="2026-10-30 22:12:01.353391144 +0000 UTC m=+1772.087754175"
Oct 30 22:12:01.393807 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:01.393762    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc64ce9a-ba29-44aa-a770-f98d29183190" path="/var/lib/kubelet/pods/cc64ce9a-ba29-44aa-a770-f98d29183190/volumes"
Oct 30 22:12:01.394331 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:01.394310    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd" path="/var/lib/kubelet/pods/f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd/volumes"
Oct 30 22:12:01.609397 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:01.609336    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl" podUID="f80c6022-402a-4810-9396-39a9765237c9" containerName="payload-processing" containerID="cri-o://a2898b29173ac8095b6acbf5b4e992200d6436beec0ad1d6b995057d80fe3f85" gracePeriod=25
Oct 30 22:12:01.890253 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:01.890225    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"
Oct 30 22:12:02.043368 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:02.043337    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f80c6022-402a-4810-9396-39a9765237c9-kube-api-access-nfjtj\" (UniqueName: \"kubernetes.io/projected/f80c6022-402a-4810-9396-39a9765237c9-kube-api-access-nfjtj\") pod \"f80c6022-402a-4810-9396-39a9765237c9\" (UID: \"f80c6022-402a-4810-9396-39a9765237c9\") "
Oct 30 22:12:02.043559 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:02.043395    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f80c6022-402a-4810-9396-39a9765237c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80c6022-402a-4810-9396-39a9765237c9-plugins-config-volume\") pod \"f80c6022-402a-4810-9396-39a9765237c9\" (UID: \"f80c6022-402a-4810-9396-39a9765237c9\") "
Oct 30 22:12:02.043771 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:02.043747    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80c6022-402a-4810-9396-39a9765237c9-plugins-config-volume" pod "f80c6022-402a-4810-9396-39a9765237c9" (UID: "f80c6022-402a-4810-9396-39a9765237c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:02.045796 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:02.045760    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80c6022-402a-4810-9396-39a9765237c9-kube-api-access-nfjtj" pod "f80c6022-402a-4810-9396-39a9765237c9" (UID: "f80c6022-402a-4810-9396-39a9765237c9"). InnerVolumeSpecName "kube-api-access-nfjtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:02.144706 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:02.144624    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfjtj\" (UniqueName: \"kubernetes.io/projected/f80c6022-402a-4810-9396-39a9765237c9-kube-api-access-nfjtj\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:02.144706 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:02.144660    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80c6022-402a-4810-9396-39a9765237c9-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:02.231301 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:02.231267    4553 generic.go:356] "Generic (PLEG): container finished" podID="f80c6022-402a-4810-9396-39a9765237c9" containerID="a2898b29173ac8095b6acbf5b4e992200d6436beec0ad1d6b995057d80fe3f85" exitCode=0
Oct 30 22:12:02.231815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:02.231315    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"
Oct 30 22:12:02.231815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:02.231359    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl" event={"ID":"f80c6022-402a-4810-9396-39a9765237c9","Type":"ContainerDied","Data":"a2898b29173ac8095b6acbf5b4e992200d6436beec0ad1d6b995057d80fe3f85"}
Oct 30 22:12:02.231815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:02.231427    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl" event={"ID":"f80c6022-402a-4810-9396-39a9765237c9","Type":"ContainerDied","Data":"1e731010b7a695d415e9dfe0e2d7d198de79cc51aece5ac963171f07e90de5b8"}
Oct 30 22:12:02.231815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:02.231459    4553 scope.go:122] "RemoveContainer" containerID="a2898b29173ac8095b6acbf5b4e992200d6436beec0ad1d6b995057d80fe3f85"
Oct 30 22:12:02.241671 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:02.241647    4553 scope.go:122] "RemoveContainer" containerID="a2898b29173ac8095b6acbf5b4e992200d6436beec0ad1d6b995057d80fe3f85"
Oct 30 22:12:02.241957 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:12:02.241939    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2898b29173ac8095b6acbf5b4e992200d6436beec0ad1d6b995057d80fe3f85\": container with ID starting with a2898b29173ac8095b6acbf5b4e992200d6436beec0ad1d6b995057d80fe3f85 not found: ID does not exist" containerID="a2898b29173ac8095b6acbf5b4e992200d6436beec0ad1d6b995057d80fe3f85"
Oct 30 22:12:02.242015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:02.241966    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2898b29173ac8095b6acbf5b4e992200d6436beec0ad1d6b995057d80fe3f85"} err="failed to get container status \"a2898b29173ac8095b6acbf5b4e992200d6436beec0ad1d6b995057d80fe3f85\": rpc error: code = NotFound desc = could not find container \"a2898b29173ac8095b6acbf5b4e992200d6436beec0ad1d6b995057d80fe3f85\": container with ID starting with a2898b29173ac8095b6acbf5b4e992200d6436beec0ad1d6b995057d80fe3f85 not found: ID does not exist"
Oct 30 22:12:02.402032 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:02.401934    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"]
Oct 30 22:12:02.451822 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:02.451789    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-479363-96bb58d89-lv8jl"]
Oct 30 22:12:03.132357 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:03.132322    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"
Oct 30 22:12:03.132644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:03.132623    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"
Oct 30 22:12:03.394683 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:03.394605    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f80c6022-402a-4810-9396-39a9765237c9" path="/var/lib/kubelet/pods/f80c6022-402a-4810-9396-39a9765237c9/volumes"
Oct 30 22:12:07.893965 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.893917    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"]
Oct 30 22:12:07.895731 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.895662    4553 cpu_manager.go:414] "containerMap: removing container" podUID="f80c6022-402a-4810-9396-39a9765237c9" containerName="payload-processing"
Oct 30 22:12:07.895938 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.895920    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80c6022-402a-4810-9396-39a9765237c9" containerName="payload-processing"
Oct 30 22:12:07.896050 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.896040    4553 cpu_manager.go:414] "containerMap: removing container" podUID="f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd" containerName="payload-processing"
Oct 30 22:12:07.896123 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.896113    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd" containerName="payload-processing"
Oct 30 22:12:07.896313 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.896301    4553 cpu_manager.go:414] "containerMap: removing container" podUID="cc64ce9a-ba29-44aa-a770-f98d29183190" containerName="payload-pre-processing"
Oct 30 22:12:07.896384 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.896373    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc64ce9a-ba29-44aa-a770-f98d29183190" containerName="payload-pre-processing"
Oct 30 22:12:07.896760 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.896741    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc64ce9a-ba29-44aa-a770-f98d29183190" containerName="payload-pre-processing"
Oct 30 22:12:07.896891 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.896881    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80c6022-402a-4810-9396-39a9765237c9" containerName="payload-processing"
Oct 30 22:12:07.896968 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.896959    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6de2f26-fd72-4d0d-bcfd-2a0534e3afbd" containerName="payload-processing"
Oct 30 22:12:07.903924 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.903889    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:07.919578 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.919550    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-maas-api-tls\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:07.919740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.919726    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-469sd\" (UniqueName: \"kubernetes.io/projected/ba061980-6125-4a2c-9cc8-15dfa898d08c-kube-api-access-469sd\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:07.919831 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.919815    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-metrics-certs\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:07.919879 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.919848    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba061980-6125-4a2c-9cc8-15dfa898d08c-openshift-service-ca\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:07.955656 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:07.955609    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"]
Oct 30 22:12:08.002373 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:08.002342    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7fabe91d-serving-cert\""
Oct 30 22:12:08.021078 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:08.021037    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-469sd\" (UniqueName: \"kubernetes.io/projected/ba061980-6125-4a2c-9cc8-15dfa898d08c-kube-api-access-469sd\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:08.021366 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:08.021114    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-metrics-certs\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:08.021366 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:08.021184    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba061980-6125-4a2c-9cc8-15dfa898d08c-openshift-service-ca\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:08.021366 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:08.021269    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-maas-api-tls\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:08.021548 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:12:08.021472    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-serving-cert: secret "maas-api-e2e-mt-7fabe91d-serving-cert" not found
Oct 30 22:12:08.021548 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:12:08.021547    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-maas-api-tls podName:ba061980-6125-4a2c-9cc8-15dfa898d08c nodeName:}" failed. No retries permitted until 2026-10-30 22:12:08.521524741 +0000 UTC m=+1779.255887752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-maas-api-tls") pod "maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z" (UID: "ba061980-6125-4a2c-9cc8-15dfa898d08c") : secret "maas-api-e2e-mt-7fabe91d-serving-cert" not found
Oct 30 22:12:08.022178 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:08.022120    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba061980-6125-4a2c-9cc8-15dfa898d08c-openshift-service-ca\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:08.024066 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:08.024037    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-metrics-certs\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:08.124660 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:08.124624    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-469sd\" (UniqueName: \"kubernetes.io/projected/ba061980-6125-4a2c-9cc8-15dfa898d08c-kube-api-access-469sd\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:08.534747 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:08.534715    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-maas-api-tls\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:08.537402 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:08.537372    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-maas-api-tls\") pod \"maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:08.820173 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:08.820039    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:09.240805 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:09.240773    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"]
Oct 30 22:12:09.241694 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:12:09.241662    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba061980_6125_4a2c_9cc8_15dfa898d08c.slice/crio-1e7176e2a54cea4d598224171861afd8ec73969c8298e569e41891e32896406d WatchSource:0}: Error finding container 1e7176e2a54cea4d598224171861afd8ec73969c8298e569e41891e32896406d: Status 404 returned error can't find the container with id 1e7176e2a54cea4d598224171861afd8ec73969c8298e569e41891e32896406d
Oct 30 22:12:09.279416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:09.279378    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z" event={"ID":"ba061980-6125-4a2c-9cc8-15dfa898d08c","Type":"ContainerStarted","Data":"1e7176e2a54cea4d598224171861afd8ec73969c8298e569e41891e32896406d"}
Oct 30 22:12:10.287261 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:10.287225    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z" event={"ID":"ba061980-6125-4a2c-9cc8-15dfa898d08c","Type":"ContainerStarted","Data":"ec95c581ccbb5f1883d3629c6470d4b00f96254490d10e7adac1e7666477c511"}
Oct 30 22:12:10.287766 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:10.287357    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:10.425067 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:10.424961    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z" podStartSLOduration=2.501931554 podStartE2EDuration="3.424946358s" totalImagesPullingTime="923.014804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:10.421990821 +0000 UTC m=+1781.156353853" watchObservedRunningTime="2026-10-30 22:12:10.424946358 +0000 UTC m=+1781.159309383"
Oct 30 22:12:11.225924 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:11.225892    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"
Oct 30 22:12:12.234327 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:12.234292    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"
Oct 30 22:12:16.298536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:16.298503    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:12:17.718820 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.718783    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"]
Oct 30 22:12:17.723227 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.723194    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.793165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.785161    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-859078-openshift-default-dockercfg-4wk49\""
Oct 30 22:12:17.898098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.895247    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-istio-envoy\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.898098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.895296    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxr8d\" (UniqueName: \"kubernetes.io/projected/f827754d-0840-4429-9ce1-8d63d32e6871-kube-api-access-qxr8d\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.898098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.895340    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f827754d-0840-4429-9ce1-8d63d32e6871-istio-token\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.898098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.895384    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-workload-socket\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.898098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.895431    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f827754d-0840-4429-9ce1-8d63d32e6871-istio-podinfo\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.898098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.895477    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-istio-data\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.898098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.895524    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f827754d-0840-4429-9ce1-8d63d32e6871-istiod-ca-cert\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.898098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.895559    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-credential-socket\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.898098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.895610    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-workload-certs\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.907488 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.907406    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"]
Oct 30 22:12:17.996844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.996758    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-istio-data\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.996844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.996814    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f827754d-0840-4429-9ce1-8d63d32e6871-istiod-ca-cert\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.996844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.996843    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-credential-socket\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.997923 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.996880    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-workload-certs\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.997923 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.996956    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-istio-envoy\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.997923 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.996981    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxr8d\" (UniqueName: \"kubernetes.io/projected/f827754d-0840-4429-9ce1-8d63d32e6871-kube-api-access-qxr8d\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.997923 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.997026    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f827754d-0840-4429-9ce1-8d63d32e6871-istio-token\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.997923 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.997075    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-workload-socket\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.997923 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.997129    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f827754d-0840-4429-9ce1-8d63d32e6871-istio-podinfo\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.997923 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.997479    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-workload-certs\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.997923 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.997489    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-workload-socket\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.997923 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.997506    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-istio-data\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.997923 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.997820    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f827754d-0840-4429-9ce1-8d63d32e6871-istiod-ca-cert\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.998466 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.998160    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-credential-socket\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:17.999822 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:17.999801    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-istio-envoy\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:18.000418 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:18.000401    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f827754d-0840-4429-9ce1-8d63d32e6871-istio-podinfo\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:18.056033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:18.056002    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f827754d-0840-4429-9ce1-8d63d32e6871-istio-token\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:18.056249 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:18.056058    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxr8d\" (UniqueName: \"kubernetes.io/projected/f827754d-0840-4429-9ce1-8d63d32e6871-kube-api-access-qxr8d\") pod \"e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") " pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:18.338109 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:18.338078    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:18.788170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:18.788071    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"]
Oct 30 22:12:18.816821 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:12:18.816781    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf827754d_0840_4429_9ce1_8d63d32e6871.slice/crio-f096871f2f2d6c9028fc66f069e623af7dcd2106d041d61fb2db52ee6964f2f5 WatchSource:0}: Error finding container f096871f2f2d6c9028fc66f069e623af7dcd2106d041d61fb2db52ee6964f2f5: Status 404 returned error can't find the container with id f096871f2f2d6c9028fc66f069e623af7dcd2106d041d61fb2db52ee6964f2f5
Oct 30 22:12:18.819389 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:18.819351    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:12:18.819469 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:18.819430    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:12:18.819469 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:18.819459    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:12:19.345349 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:19.345311    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc" event={"ID":"f827754d-0840-4429-9ce1-8d63d32e6871","Type":"ContainerStarted","Data":"619c55ba101ea597fe9d68698c30cb10e208dadf3aa152d3b398174fecef4b96"}
Oct 30 22:12:19.345551 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:19.345416    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc" event={"ID":"f827754d-0840-4429-9ce1-8d63d32e6871","Type":"ContainerStarted","Data":"f096871f2f2d6c9028fc66f069e623af7dcd2106d041d61fb2db52ee6964f2f5"}
Oct 30 22:12:19.465112 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:19.465059    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc" podStartSLOduration=2.465046107 podStartE2EDuration="2.465046107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:19.464277314 +0000 UTC m=+1790.198640386" watchObservedRunningTime="2026-10-30 22:12:19.465046107 +0000 UTC m=+1790.199409138"
Oct 30 22:12:20.338302 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:20.338269    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:20.344852 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:20.344826    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:20.351813 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:20.351789    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:20.352719 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:20.352702    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:12:26.608085 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.608044    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"]
Oct 30 22:12:26.613473 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.613449    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"
Oct 30 22:12:26.668194 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.668154    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"]
Oct 30 22:12:26.723023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.721740    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-859078-dockercfg-qkh6g\""
Oct 30 22:12:26.723023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.721850    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-859078\""
Oct 30 22:12:26.807207 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.807168    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"]
Oct 30 22:12:26.815323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.814529    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"
Oct 30 22:12:26.832270 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.832234    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"]
Oct 30 22:12:26.835966 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.835931    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntcz5\" (UniqueName: \"kubernetes.io/projected/54fc71a0-ea97-4740-8f06-d41584d804dc-kube-api-access-ntcz5\") pod \"payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g\" (UID: \"54fc71a0-ea97-4740-8f06-d41584d804dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"
Oct 30 22:12:26.836170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.836056    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54fc71a0-ea97-4740-8f06-d41584d804dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g\" (UID: \"54fc71a0-ea97-4740-8f06-d41584d804dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"
Oct 30 22:12:26.937366 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.937277    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gljjl\" (UniqueName: \"kubernetes.io/projected/2d913198-413c-496e-b54f-a4823f53df45-kube-api-access-gljjl\") pod \"payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm\" (UID: \"2d913198-413c-496e-b54f-a4823f53df45\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"
Oct 30 22:12:26.937532 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.937510    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntcz5\" (UniqueName: \"kubernetes.io/projected/54fc71a0-ea97-4740-8f06-d41584d804dc-kube-api-access-ntcz5\") pod \"payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g\" (UID: \"54fc71a0-ea97-4740-8f06-d41584d804dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"
Oct 30 22:12:26.937576 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.937556    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d913198-413c-496e-b54f-a4823f53df45-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm\" (UID: \"2d913198-413c-496e-b54f-a4823f53df45\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"
Oct 30 22:12:26.937609 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.937581    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54fc71a0-ea97-4740-8f06-d41584d804dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g\" (UID: \"54fc71a0-ea97-4740-8f06-d41584d804dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"
Oct 30 22:12:26.938313 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.938296    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54fc71a0-ea97-4740-8f06-d41584d804dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g\" (UID: \"54fc71a0-ea97-4740-8f06-d41584d804dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"
Oct 30 22:12:26.996263 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:26.996228    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntcz5\" (UniqueName: \"kubernetes.io/projected/54fc71a0-ea97-4740-8f06-d41584d804dc-kube-api-access-ntcz5\") pod \"payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g\" (UID: \"54fc71a0-ea97-4740-8f06-d41584d804dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"
Oct 30 22:12:27.038596 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:27.038557    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d913198-413c-496e-b54f-a4823f53df45-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm\" (UID: \"2d913198-413c-496e-b54f-a4823f53df45\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"
Oct 30 22:12:27.038763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:27.038622    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gljjl\" (UniqueName: \"kubernetes.io/projected/2d913198-413c-496e-b54f-a4823f53df45-kube-api-access-gljjl\") pod \"payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm\" (UID: \"2d913198-413c-496e-b54f-a4823f53df45\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"
Oct 30 22:12:27.039316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:27.039296    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d913198-413c-496e-b54f-a4823f53df45-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm\" (UID: \"2d913198-413c-496e-b54f-a4823f53df45\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"
Oct 30 22:12:27.104950 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:27.104918    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gljjl\" (UniqueName: \"kubernetes.io/projected/2d913198-413c-496e-b54f-a4823f53df45-kube-api-access-gljjl\") pod \"payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm\" (UID: \"2d913198-413c-496e-b54f-a4823f53df45\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"
Oct 30 22:12:27.134169 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:27.134005    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"
Oct 30 22:12:27.228643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:27.228604    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"
Oct 30 22:12:27.535359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:27.535331    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"]
Oct 30 22:12:27.536367 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:12:27.536342    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d913198_413c_496e_b54f_a4823f53df45.slice/crio-829e469e7f53dcf507d9a65b24e60068a03ffb8b13b9744cea10a494b2864bcb WatchSource:0}: Error finding container 829e469e7f53dcf507d9a65b24e60068a03ffb8b13b9744cea10a494b2864bcb: Status 404 returned error can't find the container with id 829e469e7f53dcf507d9a65b24e60068a03ffb8b13b9744cea10a494b2864bcb
Oct 30 22:12:27.653845 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:27.653181    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"]
Oct 30 22:12:27.738774 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:12:27.726983    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fc71a0_ea97_4740_8f06_d41584d804dc.slice/crio-89b7491dfcef017847a7acd3b0bc0efc7a683de5cc2bf268066d6d6cc2ac6432 WatchSource:0}: Error finding container 89b7491dfcef017847a7acd3b0bc0efc7a683de5cc2bf268066d6d6cc2ac6432: Status 404 returned error can't find the container with id 89b7491dfcef017847a7acd3b0bc0efc7a683de5cc2bf268066d6d6cc2ac6432
Oct 30 22:12:28.202473 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:12:28.202433    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:12:28.419436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:28.419394    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g" event={"ID":"54fc71a0-ea97-4740-8f06-d41584d804dc","Type":"ContainerStarted","Data":"de97188224fae6b41f3121b2e988d5bd25a66c1a09d9761d21d7cd1cf78936f7"}
Oct 30 22:12:28.419615 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:28.419461    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g" event={"ID":"54fc71a0-ea97-4740-8f06-d41584d804dc","Type":"ContainerStarted","Data":"89b7491dfcef017847a7acd3b0bc0efc7a683de5cc2bf268066d6d6cc2ac6432"}
Oct 30 22:12:28.419615 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:28.419507    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"
Oct 30 22:12:28.421206 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:28.421182    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm" event={"ID":"2d913198-413c-496e-b54f-a4823f53df45","Type":"ContainerStarted","Data":"fb968ab314ed0693e95cad89e49c6e7c02c7eee45ea6eab88689818b14509c74"}
Oct 30 22:12:28.421313 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:28.421222    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm" event={"ID":"2d913198-413c-496e-b54f-a4823f53df45","Type":"ContainerStarted","Data":"829e469e7f53dcf507d9a65b24e60068a03ffb8b13b9744cea10a494b2864bcb"}
Oct 30 22:12:28.421313 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:28.421244    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"
Oct 30 22:12:28.539164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:28.539042    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g" podStartSLOduration=2.539029131 podStartE2EDuration="2.539029131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:28.536981268 +0000 UTC m=+1799.271344370" watchObservedRunningTime="2026-10-30 22:12:28.539029131 +0000 UTC m=+1799.273392159"
Oct 30 22:12:28.670948 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:28.669601    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm" podStartSLOduration=2.669579823 podStartE2EDuration="2.669579823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:28.665853071 +0000 UTC m=+1799.400216144" watchObservedRunningTime="2026-10-30 22:12:28.669579823 +0000 UTC m=+1799.403942855"
Oct 30 22:12:29.705390 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:29.705359    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:12:29.707346 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:29.707323    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:12:39.428944 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:39.428906    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"
Oct 30 22:12:39.429708 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:39.429689    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"
Oct 30 22:12:54.547163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.547118    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"]
Oct 30 22:12:54.551290 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.551266    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.596931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.596890    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"]
Oct 30 22:12:54.602919 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.602885    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.623798 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.623762    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.623952 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.623804    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.623952 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.623854    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a268712-b551-4bdd-8fbd-70e124508a44-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.623952 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.623937    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a268712-b551-4bdd-8fbd-70e124508a44-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.624079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.624001    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.624079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.624044    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.624079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.624060    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.624286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.624078    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a268712-b551-4bdd-8fbd-70e124508a44-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.624286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.624096    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8n7h\" (UniqueName: \"kubernetes.io/projected/7a268712-b551-4bdd-8fbd-70e124508a44-kube-api-access-g8n7h\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.624286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.624275    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwp84\" (UniqueName: \"kubernetes.io/projected/6f396aaf-e41b-49c4-8734-cd729fa95810-kube-api-access-zwp84\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.624467 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.624312    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.624467 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.624339    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.624467 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.624416    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.624575 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.624558    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.624620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.624588    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.624620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.624608    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.624707 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.624622    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f396aaf-e41b-49c4-8734-cd729fa95810-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.624747 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.624725    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.637247 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.637220    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7fabe91d-openshift-default-dockercfg-rt9jz\""
Oct 30 22:12:54.648631 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.648601    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"]
Oct 30 22:12:54.730922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.730891    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a268712-b551-4bdd-8fbd-70e124508a44-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.730922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.730931    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8n7h\" (UniqueName: \"kubernetes.io/projected/7a268712-b551-4bdd-8fbd-70e124508a44-kube-api-access-g8n7h\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.731226 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.730984    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwp84\" (UniqueName: \"kubernetes.io/projected/6f396aaf-e41b-49c4-8734-cd729fa95810-kube-api-access-zwp84\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.731226 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731008    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.731226 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731033    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.731226 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731083    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.731447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731264    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.731447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731311    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.731447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731344    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.731447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731368    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f396aaf-e41b-49c4-8734-cd729fa95810-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.731447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731413    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.731698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731460    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.731698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731489    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.731698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731520    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a268712-b551-4bdd-8fbd-70e124508a44-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.731698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731547    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a268712-b551-4bdd-8fbd-70e124508a44-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.731698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731575    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.731698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731636    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.731698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731655    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.732102 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.731664    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.732213 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.732183    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.732308 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.732247    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.732523 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.732502    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.732775 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.732756    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.733694 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.733669    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f396aaf-e41b-49c4-8734-cd729fa95810-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.733805 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.733764    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.734007 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.733985    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.734279 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.734251    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.734469 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.734447    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a268712-b551-4bdd-8fbd-70e124508a44-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.736553 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.736529    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.736854 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.736834    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.737078 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.737060    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"]
Oct 30 22:12:54.737609 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.737588    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.738233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.738201    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a268712-b551-4bdd-8fbd-70e124508a44-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.799544 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.799455    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.799544 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.799482    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a268712-b551-4bdd-8fbd-70e124508a44-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.799759 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.799565    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwp84\" (UniqueName: \"kubernetes.io/projected/6f396aaf-e41b-49c4-8734-cd729fa95810-kube-api-access-zwp84\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.799759 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.799701    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8n7h\" (UniqueName: \"kubernetes.io/projected/7a268712-b551-4bdd-8fbd-70e124508a44-kube-api-access-g8n7h\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:54.866213 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.866179    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:54.922698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:54.922459    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:55.309054 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.309023    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"]
Oct 30 22:12:55.342725 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:12:55.342694    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f396aaf_e41b_49c4_8734_cd729fa95810.slice/crio-2c705d9fae57b617483104a3ac90094ffe29579e841ae98acb608864a996cde1 WatchSource:0}: Error finding container 2c705d9fae57b617483104a3ac90094ffe29579e841ae98acb608864a996cde1: Status 404 returned error can't find the container with id 2c705d9fae57b617483104a3ac90094ffe29579e841ae98acb608864a996cde1
Oct 30 22:12:55.345220 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.345185    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:12:55.345340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.345275    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:12:55.345340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.345323    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:12:55.383468 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.383438    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"]
Oct 30 22:12:55.384732 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:12:55.384705    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a268712_b551_4bdd_8fbd_70e124508a44.slice/crio-99ef99ea82340ff1da68b0517adc1c1e7940167aeb0166988351b0b99b9ddcd9 WatchSource:0}: Error finding container 99ef99ea82340ff1da68b0517adc1c1e7940167aeb0166988351b0b99b9ddcd9: Status 404 returned error can't find the container with id 99ef99ea82340ff1da68b0517adc1c1e7940167aeb0166988351b0b99b9ddcd9
Oct 30 22:12:55.387544 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.387509    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:12:55.387644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.387578    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:12:55.387644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.387607    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:12:55.627664 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.627627    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg" event={"ID":"7a268712-b551-4bdd-8fbd-70e124508a44","Type":"ContainerStarted","Data":"0066d0c30e041d40f3e01d85ebb13ac339ef9a598c461ce89f91bee141afaa93"}
Oct 30 22:12:55.628200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.627685    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg" event={"ID":"7a268712-b551-4bdd-8fbd-70e124508a44","Type":"ContainerStarted","Data":"99ef99ea82340ff1da68b0517adc1c1e7940167aeb0166988351b0b99b9ddcd9"}
Oct 30 22:12:55.629704 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.629664    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm" event={"ID":"6f396aaf-e41b-49c4-8734-cd729fa95810","Type":"ContainerStarted","Data":"16b86f83287a350cde6c678e9ed71beebe709a039a7ba233af6fcb76b5ef282a"}
Oct 30 22:12:55.629704 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.629708    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm" event={"ID":"6f396aaf-e41b-49c4-8734-cd729fa95810","Type":"ContainerStarted","Data":"2c705d9fae57b617483104a3ac90094ffe29579e841ae98acb608864a996cde1"}
Oct 30 22:12:55.771945 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.771880    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg" podStartSLOduration=1.7718597109999998 podStartE2EDuration="1.771859711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:55.767712387 +0000 UTC m=+1826.502075420" watchObservedRunningTime="2026-10-30 22:12:55.771859711 +0000 UTC m=+1826.506222742"
Oct 30 22:12:55.866993 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.866957    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:12:55.902997 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.902872    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm" podStartSLOduration=1.902852029 podStartE2EDuration="1.902852029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:55.901368706 +0000 UTC m=+1826.635731729" watchObservedRunningTime="2026-10-30 22:12:55.902852029 +0000 UTC m=+1826.637215060"
Oct 30 22:12:55.927159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:55.926233    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:12:56.870162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:56.869282    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:12:56.870162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:56.869336    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm" podUID="6f396aaf-e41b-49c4-8734-cd729fa95810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 22:12:56.931159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:56.927275    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:12:56.931159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:56.927342    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg" podUID="7a268712-b551-4bdd-8fbd-70e124508a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 22:12:57.866671 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:57.866629    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:12:57.866863 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:57.866688    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm" podUID="6f396aaf-e41b-49c4-8734-cd729fa95810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 22:12:57.926167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:57.924618    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:12:57.926167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:57.924684    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg" podUID="7a268712-b551-4bdd-8fbd-70e124508a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 22:12:58.206760 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:12:58.204001    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:12:58.867462 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:58.866973    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:12:58.867462 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:58.867050    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm" podUID="6f396aaf-e41b-49c4-8734-cd729fa95810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 22:12:58.940683 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:58.940646    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:12:58.941330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:58.941290    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg" podUID="7a268712-b551-4bdd-8fbd-70e124508a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 22:12:59.866679 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:59.866647    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:12:59.866979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:59.866706    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm" podUID="6f396aaf-e41b-49c4-8734-cd729fa95810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 22:12:59.929163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:59.924607    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:12:59.929163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:12:59.924695    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg" podUID="7a268712-b551-4bdd-8fbd-70e124508a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 22:13:00.870219 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:00.869522    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:13:00.870219 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:00.869583    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm" podUID="6f396aaf-e41b-49c4-8734-cd729fa95810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 22:13:00.934158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:00.929485    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:13:00.934158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:00.929546    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg" podUID="7a268712-b551-4bdd-8fbd-70e124508a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 22:13:01.880162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:01.875563    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:13:01.880162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:01.875650    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm" podUID="6f396aaf-e41b-49c4-8734-cd729fa95810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Oct 30 22:13:01.924277 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:01.924234    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body=
Oct 30 22:13:01.924484 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:01.924305    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg" podUID="7a268712-b551-4bdd-8fbd-70e124508a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused"
Oct 30 22:13:02.872104 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:02.872066    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:13:02.872531 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:02.872506    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:13:02.873339 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:02.873319    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:13:02.927963 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:02.927933    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:13:02.928415 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:02.928218    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:13:02.928917 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:02.928896    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:13:06.384496 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.384464    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"]
Oct 30 22:13:06.384979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.384672    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m" podUID="6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28" containerName="maas-api" containerID="cri-o://6bdb0884e5db2ba9f7be04b3c3b0bdd72785a7b25806270b7fad2cc986aca43c" gracePeriod=30
Oct 30 22:13:06.702831 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.702506    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:13:06.732728 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.732691    4553 generic.go:356] "Generic (PLEG): container finished" podID="6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28" containerID="6bdb0884e5db2ba9f7be04b3c3b0bdd72785a7b25806270b7fad2cc986aca43c" exitCode=0
Oct 30 22:13:06.732925 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.732746    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"
Oct 30 22:13:06.732925 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.732778    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m" event={"ID":"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28","Type":"ContainerDied","Data":"6bdb0884e5db2ba9f7be04b3c3b0bdd72785a7b25806270b7fad2cc986aca43c"}
Oct 30 22:13:06.732925 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.732836    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m" event={"ID":"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28","Type":"ContainerDied","Data":"89f2dc124fdb273491d003a5e142d2093fb66305ace2df1f74ed0e9f7608d876"}
Oct 30 22:13:06.732925 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.732863    4553 scope.go:122] "RemoveContainer" containerID="6bdb0884e5db2ba9f7be04b3c3b0bdd72785a7b25806270b7fad2cc986aca43c"
Oct 30 22:13:06.743124 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.743102    4553 scope.go:122] "RemoveContainer" containerID="6bdb0884e5db2ba9f7be04b3c3b0bdd72785a7b25806270b7fad2cc986aca43c"
Oct 30 22:13:06.743411 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:13:06.743392    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bdb0884e5db2ba9f7be04b3c3b0bdd72785a7b25806270b7fad2cc986aca43c\": container with ID starting with 6bdb0884e5db2ba9f7be04b3c3b0bdd72785a7b25806270b7fad2cc986aca43c not found: ID does not exist" containerID="6bdb0884e5db2ba9f7be04b3c3b0bdd72785a7b25806270b7fad2cc986aca43c"
Oct 30 22:13:06.743481 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.743418    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bdb0884e5db2ba9f7be04b3c3b0bdd72785a7b25806270b7fad2cc986aca43c"} err="failed to get container status \"6bdb0884e5db2ba9f7be04b3c3b0bdd72785a7b25806270b7fad2cc986aca43c\": rpc error: code = NotFound desc = could not find container \"6bdb0884e5db2ba9f7be04b3c3b0bdd72785a7b25806270b7fad2cc986aca43c\": container with ID starting with 6bdb0884e5db2ba9f7be04b3c3b0bdd72785a7b25806270b7fad2cc986aca43c not found: ID does not exist"
Oct 30 22:13:06.796014 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.795981    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-openshift-service-ca\") pod \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") "
Oct 30 22:13:06.796014 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.796018    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-kube-api-access-s7m7m\" (UniqueName: \"kubernetes.io/projected/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-kube-api-access-s7m7m\") pod \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") "
Oct 30 22:13:06.796284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.796040    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-metrics-certs\") pod \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") "
Oct 30 22:13:06.796284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.796076    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-maas-api-tls\") pod \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\" (UID: \"6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28\") "
Oct 30 22:13:06.796585 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.796552    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-openshift-service-ca" pod "6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28" (UID: "6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:06.798979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.798952    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-maas-api-tls" pod "6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28" (UID: "6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:06.799079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.798982    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-metrics-certs" pod "6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28" (UID: "6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:06.799079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.798998    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-kube-api-access-s7m7m" pod "6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28" (UID: "6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28"). InnerVolumeSpecName "kube-api-access-s7m7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:06.900911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.898768    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:06.900911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.898812    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7m7m\" (UniqueName: \"kubernetes.io/projected/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-kube-api-access-s7m7m\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:06.900911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.898832    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:06.900911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:06.898842    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:07.158914 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:07.158829    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"]
Oct 30 22:13:07.164633 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:07.164597    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d73ad7-679cc9c88-q8v8m"]
Oct 30 22:13:07.399446 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:07.399405    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28" path="/var/lib/kubelet/pods/6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28/volumes"
Oct 30 22:13:07.497229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:07.496411    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"]
Oct 30 22:13:07.497229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:07.496685    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq" podUID="494ed239-bcbe-4ba4-a79d-a46b23d62456" containerName="istio-proxy" containerID="cri-o://bc6710ec824a290d29c53fe6b902a583152eaa543339acfd91b1151e43ee1089" gracePeriod=30
Oct 30 22:13:09.670810 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.670687    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"]
Oct 30 22:13:09.671361 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.671345    4553 cpu_manager.go:414] "containerMap: removing container" podUID="6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28" containerName="maas-api"
Oct 30 22:13:09.671413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.671363    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28" containerName="maas-api"
Oct 30 22:13:09.671494 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.671485    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5c81a1-c0f5-4c31-b1c5-ddd80a270e28" containerName="maas-api"
Oct 30 22:13:09.676346 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.676325    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.762656 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.762618    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.762836 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.762678    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.762836 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.762814    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz5nh\" (UniqueName: \"kubernetes.io/projected/a765ef62-b023-42d6-96c1-8e7715f11f3c-kube-api-access-nz5nh\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.762917 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.762840    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.762917 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.762866    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.762996 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.762960    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.763031 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.763012    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a765ef62-b023-42d6-96c1-8e7715f11f3c-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.763208 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.763173    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.763295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.763279    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.864834 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.864803    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.865009 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.864918    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz5nh\" (UniqueName: \"kubernetes.io/projected/a765ef62-b023-42d6-96c1-8e7715f11f3c-kube-api-access-nz5nh\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.865009 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.864956    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.865009 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.864991    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.865201 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.865045    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.865201 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.865122    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a765ef62-b023-42d6-96c1-8e7715f11f3c-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.865321 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.865217    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.865321 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.865269    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.865321 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.865297    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-data\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.865321 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.865317    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.866151 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.866110    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a765ef62-b023-42d6-96c1-8e7715f11f3c-istiod-ca-cert\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.866331 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.866235    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-credential-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.866478 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.866450    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-workload-socket\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.867062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.867033    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-workload-certs\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.868325 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.868298    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-envoy\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.868467 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.868451    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-podinfo\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.900305 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.900204    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"]
Oct 30 22:13:09.925225 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.923738    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-token\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.925225 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.924180    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz5nh\" (UniqueName: \"kubernetes.io/projected/a765ef62-b023-42d6-96c1-8e7715f11f3c-kube-api-access-nz5nh\") pod \"e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") " pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:09.991279 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:09.991245    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:10.426053 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:10.426021    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"]
Oct 30 22:13:10.427246 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:13:10.427199    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda765ef62_b023_42d6_96c1_8e7715f11f3c.slice/crio-61ccc6c7244def3e4c71bc4998b6b73dc741a7e6d93f5420009c3457300e8eb7 WatchSource:0}: Error finding container 61ccc6c7244def3e4c71bc4998b6b73dc741a7e6d93f5420009c3457300e8eb7: Status 404 returned error can't find the container with id 61ccc6c7244def3e4c71bc4998b6b73dc741a7e6d93f5420009c3457300e8eb7
Oct 30 22:13:10.430007 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:10.429979    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:10.430088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:10.430046    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:10.430126 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:10.430097    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:10.763267 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:10.763167    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv" event={"ID":"a765ef62-b023-42d6-96c1-8e7715f11f3c","Type":"ContainerStarted","Data":"6ef6f7b7651322d793e73addc4d47914d03c8ca947e9ae38d6f158fcd82be4f9"}
Oct 30 22:13:10.763267 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:10.763229    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv" event={"ID":"a765ef62-b023-42d6-96c1-8e7715f11f3c","Type":"ContainerStarted","Data":"61ccc6c7244def3e4c71bc4998b6b73dc741a7e6d93f5420009c3457300e8eb7"}
Oct 30 22:13:10.992055 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:10.992021    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:11.993678 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:11.993635    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:13:11.994191 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:11.993692    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv" podUID="a765ef62-b023-42d6-96c1-8e7715f11f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:13:12.768700 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.768674    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:13:12.780587 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.780559    4553 generic.go:356] "Generic (PLEG): container finished" podID="494ed239-bcbe-4ba4-a79d-a46b23d62456" containerID="bc6710ec824a290d29c53fe6b902a583152eaa543339acfd91b1151e43ee1089" exitCode=0
Oct 30 22:13:12.780747 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.780607    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"
Oct 30 22:13:12.780747 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.780643    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq" event={"ID":"494ed239-bcbe-4ba4-a79d-a46b23d62456","Type":"ContainerDied","Data":"bc6710ec824a290d29c53fe6b902a583152eaa543339acfd91b1151e43ee1089"}
Oct 30 22:13:12.780747 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.780696    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq" event={"ID":"494ed239-bcbe-4ba4-a79d-a46b23d62456","Type":"ContainerDied","Data":"a6bbe0cf2764362d153c8d62121f152561f8490aa0adbb475ea0a498afe96a62"}
Oct 30 22:13:12.780747 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.780724    4553 scope.go:122] "RemoveContainer" containerID="bc6710ec824a290d29c53fe6b902a583152eaa543339acfd91b1151e43ee1089"
Oct 30 22:13:12.790957 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.790938    4553 scope.go:122] "RemoveContainer" containerID="bc6710ec824a290d29c53fe6b902a583152eaa543339acfd91b1151e43ee1089"
Oct 30 22:13:12.791276 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:13:12.791258    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6710ec824a290d29c53fe6b902a583152eaa543339acfd91b1151e43ee1089\": container with ID starting with bc6710ec824a290d29c53fe6b902a583152eaa543339acfd91b1151e43ee1089 not found: ID does not exist" containerID="bc6710ec824a290d29c53fe6b902a583152eaa543339acfd91b1151e43ee1089"
Oct 30 22:13:12.791324 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.791288    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6710ec824a290d29c53fe6b902a583152eaa543339acfd91b1151e43ee1089"} err="failed to get container status \"bc6710ec824a290d29c53fe6b902a583152eaa543339acfd91b1151e43ee1089\": rpc error: code = NotFound desc = could not find container \"bc6710ec824a290d29c53fe6b902a583152eaa543339acfd91b1151e43ee1089\": container with ID starting with bc6710ec824a290d29c53fe6b902a583152eaa543339acfd91b1151e43ee1089 not found: ID does not exist"
Oct 30 22:13:12.801684 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.801660    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-envoy\") pod \"494ed239-bcbe-4ba4-a79d-a46b23d62456\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") "
Oct 30 22:13:12.801815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.801698    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494ed239-bcbe-4ba4-a79d-a46b23d62456-kube-api-access-9zjcl\" (UniqueName: \"kubernetes.io/projected/494ed239-bcbe-4ba4-a79d-a46b23d62456-kube-api-access-9zjcl\") pod \"494ed239-bcbe-4ba4-a79d-a46b23d62456\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") "
Oct 30 22:13:12.801869 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.801813    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-credential-socket\") pod \"494ed239-bcbe-4ba4-a79d-a46b23d62456\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") "
Oct 30 22:13:12.801923 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.801879    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-podinfo\") pod \"494ed239-bcbe-4ba4-a79d-a46b23d62456\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") "
Oct 30 22:13:12.801976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.801917    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-token\" (UniqueName: \"kubernetes.io/projected/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-token\") pod \"494ed239-bcbe-4ba4-a79d-a46b23d62456\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") "
Oct 30 22:13:12.802049 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.802034    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-data\") pod \"494ed239-bcbe-4ba4-a79d-a46b23d62456\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") "
Oct 30 22:13:12.802111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.802099    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-workload-socket\") pod \"494ed239-bcbe-4ba4-a79d-a46b23d62456\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") "
Oct 30 22:13:12.802181 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.802129    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-workload-certs\") pod \"494ed239-bcbe-4ba4-a79d-a46b23d62456\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") "
Oct 30 22:13:12.802181 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.802175    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494ed239-bcbe-4ba4-a79d-a46b23d62456-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494ed239-bcbe-4ba4-a79d-a46b23d62456-istiod-ca-cert\") pod \"494ed239-bcbe-4ba4-a79d-a46b23d62456\" (UID: \"494ed239-bcbe-4ba4-a79d-a46b23d62456\") "
Oct 30 22:13:12.802496 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.802476    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-data" pod "494ed239-bcbe-4ba4-a79d-a46b23d62456" (UID: "494ed239-bcbe-4ba4-a79d-a46b23d62456"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:12.802567 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.802439    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-credential-socket" pod "494ed239-bcbe-4ba4-a79d-a46b23d62456" (UID: "494ed239-bcbe-4ba4-a79d-a46b23d62456"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:12.802947 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.802926    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:12.803019 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.802954    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:12.803454 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.803214    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-workload-socket" pod "494ed239-bcbe-4ba4-a79d-a46b23d62456" (UID: "494ed239-bcbe-4ba4-a79d-a46b23d62456"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:12.803454 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.803260    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494ed239-bcbe-4ba4-a79d-a46b23d62456-istiod-ca-cert" pod "494ed239-bcbe-4ba4-a79d-a46b23d62456" (UID: "494ed239-bcbe-4ba4-a79d-a46b23d62456"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:12.803454 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.803321    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-workload-certs" pod "494ed239-bcbe-4ba4-a79d-a46b23d62456" (UID: "494ed239-bcbe-4ba4-a79d-a46b23d62456"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:12.804704 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.804675    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-envoy" pod "494ed239-bcbe-4ba4-a79d-a46b23d62456" (UID: "494ed239-bcbe-4ba4-a79d-a46b23d62456"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:12.804704 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.804684    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494ed239-bcbe-4ba4-a79d-a46b23d62456-kube-api-access-9zjcl" pod "494ed239-bcbe-4ba4-a79d-a46b23d62456" (UID: "494ed239-bcbe-4ba4-a79d-a46b23d62456"). InnerVolumeSpecName "kube-api-access-9zjcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:12.805014 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.804979    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-podinfo" pod "494ed239-bcbe-4ba4-a79d-a46b23d62456" (UID: "494ed239-bcbe-4ba4-a79d-a46b23d62456"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:12.805791 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.805764    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-token" pod "494ed239-bcbe-4ba4-a79d-a46b23d62456" (UID: "494ed239-bcbe-4ba4-a79d-a46b23d62456"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:12.903988 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.903954    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:12.903988 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.903986    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:12.903988 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.903998    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:12.904266 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.904010    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:12.904266 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.904023    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/494ed239-bcbe-4ba4-a79d-a46b23d62456-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:12.904266 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.904035    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/494ed239-bcbe-4ba4-a79d-a46b23d62456-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:12.904266 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.904047    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zjcl\" (UniqueName: \"kubernetes.io/projected/494ed239-bcbe-4ba4-a79d-a46b23d62456-kube-api-access-9zjcl\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:12.953541 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.953474    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv" podStartSLOduration=3.953453504 podStartE2EDuration="3.953453504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:10.932333141 +0000 UTC m=+1841.666696173" watchObservedRunningTime="2026-10-30 22:13:12.953453504 +0000 UTC m=+1843.687816535"
Oct 30 22:13:12.992099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.992056    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:13:12.992272 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:12.992127    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv" podUID="a765ef62-b023-42d6-96c1-8e7715f11f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:13:13.236263 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.236209    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"]
Oct 30 22:13:13.241866 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.241835    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-qr2zq"]
Oct 30 22:13:13.333714 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.333671    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"]
Oct 30 22:13:13.334457 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.334438    4553 cpu_manager.go:414] "containerMap: removing container" podUID="494ed239-bcbe-4ba4-a79d-a46b23d62456" containerName="istio-proxy"
Oct 30 22:13:13.334457 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.334458    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494ed239-bcbe-4ba4-a79d-a46b23d62456" containerName="istio-proxy"
Oct 30 22:13:13.334601 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.334551    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="494ed239-bcbe-4ba4-a79d-a46b23d62456" containerName="istio-proxy"
Oct 30 22:13:13.339893 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.339831    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:13.393322 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.393285    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-859078-serving-cert\""
Oct 30 22:13:13.396324 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.396296    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494ed239-bcbe-4ba4-a79d-a46b23d62456" path="/var/lib/kubelet/pods/494ed239-bcbe-4ba4-a79d-a46b23d62456/volumes"
Oct 30 22:13:13.396661 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.396645    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"]
Oct 30 22:13:13.438972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.438869    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-metrics-certs\") pod \"maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:13.439167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.438977    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:13.439167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.439105    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:13.439489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.439459    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7nxb\" (UniqueName: \"kubernetes.io/projected/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-kube-api-access-w7nxb\") pod \"maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:13.540748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.540704    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-metrics-certs\") pod \"maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:13.540954 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.540825    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:13.540954 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.540863    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:13.541063 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.541005    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7nxb\" (UniqueName: \"kubernetes.io/projected/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-kube-api-access-w7nxb\") pod \"maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:13.541514 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:13:13.541488    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-serving-cert: secret "maas-api-e2e-ipp-b-859078-serving-cert" not found
Oct 30 22:13:13.541606 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:13:13.541564    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-maas-api-tls podName:ebef8f6b-459e-4a01-8fa2-ed06d04208c5 nodeName:}" failed. No retries permitted until 2026-10-30 22:13:14.041543263 +0000 UTC m=+1844.775906276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-maas-api-tls") pod "maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl" (UID: "ebef8f6b-459e-4a01-8fa2-ed06d04208c5") : secret "maas-api-e2e-ipp-b-859078-serving-cert" not found
Oct 30 22:13:13.542078 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.542051    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:13.544480 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.544448    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-metrics-certs\") pod \"maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:13.604410 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.604335    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7nxb\" (UniqueName: \"kubernetes.io/projected/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-kube-api-access-w7nxb\") pod \"maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:13.992171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.992044    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:13:13.992171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:13.992100    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv" podUID="a765ef62-b023-42d6-96c1-8e7715f11f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:13:14.062422 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:14.062371    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:14.065109 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:14.065076    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:14.259861 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:14.259767    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:14.670664 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:14.670634    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"]
Oct 30 22:13:14.671939 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:13:14.671913    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebef8f6b_459e_4a01_8fa2_ed06d04208c5.slice/crio-4006e1c005f4c5f45ce051de2782cd23793635aa2137e4a51d4b76b2f784ad81 WatchSource:0}: Error finding container 4006e1c005f4c5f45ce051de2782cd23793635aa2137e4a51d4b76b2f784ad81: Status 404 returned error can't find the container with id 4006e1c005f4c5f45ce051de2782cd23793635aa2137e4a51d4b76b2f784ad81
Oct 30 22:13:14.841960 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:14.841923    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl" event={"ID":"ebef8f6b-459e-4a01-8fa2-ed06d04208c5","Type":"ContainerStarted","Data":"4006e1c005f4c5f45ce051de2782cd23793635aa2137e4a51d4b76b2f784ad81"}
Oct 30 22:13:14.992658 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:14.992569    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:13:14.992658 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:14.992622    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv" podUID="a765ef62-b023-42d6-96c1-8e7715f11f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:13:15.881598 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:15.881561    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl" event={"ID":"ebef8f6b-459e-4a01-8fa2-ed06d04208c5","Type":"ContainerStarted","Data":"8572f73e012c549a0492c89983b5afa7d94a43c51ab93be7d2a15025051db5c3"}
Oct 30 22:13:15.882485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:15.882435    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:15.992030 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:15.992002    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:13:15.992238 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:15.992053    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv" podUID="a765ef62-b023-42d6-96c1-8e7715f11f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:13:16.002742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:16.002690    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl" podStartSLOduration=2.169355309 podStartE2EDuration="3.002676591s" totalImagesPullingTime="833.321282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:16.001536042 +0000 UTC m=+1846.735899074" watchObservedRunningTime="2026-10-30 22:13:16.002676591 +0000 UTC m=+1846.737039621"
Oct 30 22:13:17.098431 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:17.098395    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:17.099177 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:17.099117    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:17.100293 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:17.100272    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:13:22.918377 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:22.918343    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:13:25.761880 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:25.761840    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"]
Oct 30 22:13:25.873110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:25.873060    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"]
Oct 30 22:13:28.203011 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:13:28.202976    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:13:29.943960 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:29.943926    4553 scope.go:122] "RemoveContainer" containerID="0773f0d07c503dc884cbb01ecadd309ebc074d6bd503a36b40fef3910983d8da"
Oct 30 22:13:30.762832 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:30.762748    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4" podUID="e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c" containerName="payload-processing" containerID="cri-o://50c6189376b3c21b7a316d98a96914aee9f31b08be70e8db5f583d6f5fe76141" gracePeriod=25
Oct 30 22:13:30.874558 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:30.874513    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6" podUID="13c0c4e6-7049-4860-a606-4fb171299052" containerName="payload-pre-processing" containerID="cri-o://e9eb8a6212d4dd1f66885f964c19ac3f943d689bf91fa9a20de9ae211f9cd8df" gracePeriod=25
Oct 30 22:13:31.035292 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.035204    4553 generic.go:356] "Generic (PLEG): container finished" podID="e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c" containerID="50c6189376b3c21b7a316d98a96914aee9f31b08be70e8db5f583d6f5fe76141" exitCode=0
Oct 30 22:13:31.035662 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.035283    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4" event={"ID":"e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c","Type":"ContainerDied","Data":"50c6189376b3c21b7a316d98a96914aee9f31b08be70e8db5f583d6f5fe76141"}
Oct 30 22:13:31.038882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.037367    4553 generic.go:356] "Generic (PLEG): container finished" podID="13c0c4e6-7049-4860-a606-4fb171299052" containerID="e9eb8a6212d4dd1f66885f964c19ac3f943d689bf91fa9a20de9ae211f9cd8df" exitCode=0
Oct 30 22:13:31.038882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.037500    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6" event={"ID":"13c0c4e6-7049-4860-a606-4fb171299052","Type":"ContainerDied","Data":"e9eb8a6212d4dd1f66885f964c19ac3f943d689bf91fa9a20de9ae211f9cd8df"}
Oct 30 22:13:31.115804 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.115774    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"
Oct 30 22:13:31.186948 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.186921    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"
Oct 30 22:13:31.228207 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.228176    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c-plugins-config-volume\") pod \"e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c\" (UID: \"e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c\") "
Oct 30 22:13:31.228379 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.228236    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c-kube-api-access-zcfz8\" (UniqueName: \"kubernetes.io/projected/e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c-kube-api-access-zcfz8\") pod \"e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c\" (UID: \"e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c\") "
Oct 30 22:13:31.228528 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.228505    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c-plugins-config-volume" pod "e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c" (UID: "e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:31.228644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.228627    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:31.230509 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.230485    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c-kube-api-access-zcfz8" pod "e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c" (UID: "e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c"). InnerVolumeSpecName "kube-api-access-zcfz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:31.330101 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.330013    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c0c4e6-7049-4860-a606-4fb171299052-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c0c4e6-7049-4860-a606-4fb171299052-plugins-config-volume\") pod \"13c0c4e6-7049-4860-a606-4fb171299052\" (UID: \"13c0c4e6-7049-4860-a606-4fb171299052\") "
Oct 30 22:13:31.330101 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.330087    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c0c4e6-7049-4860-a606-4fb171299052-kube-api-access-7lhbq\" (UniqueName: \"kubernetes.io/projected/13c0c4e6-7049-4860-a606-4fb171299052-kube-api-access-7lhbq\") pod \"13c0c4e6-7049-4860-a606-4fb171299052\" (UID: \"13c0c4e6-7049-4860-a606-4fb171299052\") "
Oct 30 22:13:31.330493 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.330470    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c0c4e6-7049-4860-a606-4fb171299052-plugins-config-volume" pod "13c0c4e6-7049-4860-a606-4fb171299052" (UID: "13c0c4e6-7049-4860-a606-4fb171299052"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:31.330696 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.330683    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcfz8\" (UniqueName: \"kubernetes.io/projected/e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c-kube-api-access-zcfz8\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:31.330739 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.330699    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13c0c4e6-7049-4860-a606-4fb171299052-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:31.332398 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.332373    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c0c4e6-7049-4860-a606-4fb171299052-kube-api-access-7lhbq" pod "13c0c4e6-7049-4860-a606-4fb171299052" (UID: "13c0c4e6-7049-4860-a606-4fb171299052"). InnerVolumeSpecName "kube-api-access-7lhbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:31.432207 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:31.432175    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lhbq\" (UniqueName: \"kubernetes.io/projected/13c0c4e6-7049-4860-a606-4fb171299052-kube-api-access-7lhbq\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:32.058402 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:32.058307    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6" event={"ID":"13c0c4e6-7049-4860-a606-4fb171299052","Type":"ContainerDied","Data":"1535b4f4de169719d9949b5a667490a74f19d25c29058f20c1acaf3f73db8a03"}
Oct 30 22:13:32.058402 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:32.058380    4553 scope.go:122] "RemoveContainer" containerID="e9eb8a6212d4dd1f66885f964c19ac3f943d689bf91fa9a20de9ae211f9cd8df"
Oct 30 22:13:32.058950 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:32.058538    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"
Oct 30 22:13:32.061898 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:32.061439    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4" event={"ID":"e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c","Type":"ContainerDied","Data":"5002e5508ecf3a09a7936dca0878d193b3fff7539198c88c889712116819c85d"}
Oct 30 22:13:32.061898 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:32.061546    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"
Oct 30 22:13:32.080611 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:32.080583    4553 scope.go:122] "RemoveContainer" containerID="50c6189376b3c21b7a316d98a96914aee9f31b08be70e8db5f583d6f5fe76141"
Oct 30 22:13:32.225594 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:32.225557    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"]
Oct 30 22:13:32.275949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:32.275915    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7fabe91d-669dbb6fdf-2fmj4"]
Oct 30 22:13:32.343796 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:32.343759    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"]
Oct 30 22:13:32.348445 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:32.347973    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7fabe91d-77674764ff-lkfw6"]
Oct 30 22:13:33.393384 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:33.393353    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13c0c4e6-7049-4860-a606-4fb171299052" path="/var/lib/kubelet/pods/13c0c4e6-7049-4860-a606-4fb171299052/volumes"
Oct 30 22:13:33.393796 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:33.393782    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c" path="/var/lib/kubelet/pods/e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c/volumes"
Oct 30 22:13:34.131781 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.131741    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"]
Oct 30 22:13:34.132453 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.132430    4553 cpu_manager.go:414] "containerMap: removing container" podUID="13c0c4e6-7049-4860-a606-4fb171299052" containerName="payload-pre-processing"
Oct 30 22:13:34.132580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.132453    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13c0c4e6-7049-4860-a606-4fb171299052" containerName="payload-pre-processing"
Oct 30 22:13:34.132580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.132499    4553 cpu_manager.go:414] "containerMap: removing container" podUID="e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c" containerName="payload-processing"
Oct 30 22:13:34.132580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.132507    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c" containerName="payload-processing"
Oct 30 22:13:34.132741 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.132679    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="13c0c4e6-7049-4860-a606-4fb171299052" containerName="payload-pre-processing"
Oct 30 22:13:34.132741 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.132700    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="e994e348-81a2-4de2-9aa3-ca9fe3dc0a9c" containerName="payload-processing"
Oct 30 22:13:34.137930 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.137900    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.210166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.194994    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-workload-certs\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.222469 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.222434    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-envoy\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.222876 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.222852    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38824b8a-8c76-412c-9d99-eb8955f8d237-istiod-ca-cert\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.223039 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.223022    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-podinfo\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.223237 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.223197    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-workload-socket\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.223392 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.223374    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-token\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.223640 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.223623    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-data\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.223955 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.223915    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-credential-socket\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.224099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.224082    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rd5l\" (UniqueName: \"kubernetes.io/projected/38824b8a-8c76-412c-9d99-eb8955f8d237-kube-api-access-7rd5l\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.283766 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.283729    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7a54db-openshift-default-dockercfg-z87rh\""
Oct 30 22:13:34.333161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.329984    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38824b8a-8c76-412c-9d99-eb8955f8d237-istiod-ca-cert\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.333161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.330780    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38824b8a-8c76-412c-9d99-eb8955f8d237-istiod-ca-cert\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.345590 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.330919    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-podinfo\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.346255 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.346182    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-workload-socket\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.368180 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.365848    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-podinfo\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.416869 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.416715    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"]
Oct 30 22:13:34.417521 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.417488    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-workload-socket\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.417618 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.417547    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-token\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.417686 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.417672    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-data\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.417821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.417805    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-credential-socket\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.417884 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.417834    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rd5l\" (UniqueName: \"kubernetes.io/projected/38824b8a-8c76-412c-9d99-eb8955f8d237-kube-api-access-7rd5l\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.417936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.417910    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-workload-certs\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.417987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.417945    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-envoy\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.420430 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.420379    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-data\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.421624 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.421351    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-credential-socket\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.421624 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.421588    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-workload-certs\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.431200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.424221    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-envoy\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.482916 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.482882    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-token\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.487992 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.487958    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rd5l\" (UniqueName: \"kubernetes.io/projected/38824b8a-8c76-412c-9d99-eb8955f8d237-kube-api-access-7rd5l\") pod \"e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") " pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:34.758360 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:34.758247    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:35.243424 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:35.243373    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"]
Oct 30 22:13:35.250508 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:13:35.249909    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38824b8a_8c76_412c_9d99_eb8955f8d237.slice/crio-61beb54e3a85bcd2c8747c492cec9133ae65d38531d38e5e308b424b7f91ef85 WatchSource:0}: Error finding container 61beb54e3a85bcd2c8747c492cec9133ae65d38531d38e5e308b424b7f91ef85: Status 404 returned error can't find the container with id 61beb54e3a85bcd2c8747c492cec9133ae65d38531d38e5e308b424b7f91ef85
Oct 30 22:13:35.253467 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:35.253430    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:35.253561 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:35.253506    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:35.253561 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:35.253537    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:36.091236 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:36.091178    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn" event={"ID":"38824b8a-8c76-412c-9d99-eb8955f8d237","Type":"ContainerStarted","Data":"7b6b263643855f0afd3042400b79885698766cea56fed4bc4c929c90faee8d22"}
Oct 30 22:13:36.091654 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:36.091250    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn" event={"ID":"38824b8a-8c76-412c-9d99-eb8955f8d237","Type":"ContainerStarted","Data":"61beb54e3a85bcd2c8747c492cec9133ae65d38531d38e5e308b424b7f91ef85"}
Oct 30 22:13:36.759357 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:36.759316    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:36.765481 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:36.765450    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:36.905738 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:36.905654    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn" podStartSLOduration=2.905637059 podStartE2EDuration="2.905637059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:36.234494337 +0000 UTC m=+1866.968857369" watchObservedRunningTime="2026-10-30 22:13:36.905637059 +0000 UTC m=+1867.640000091"
Oct 30 22:13:37.122466 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:37.122434    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:37.129404 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:37.129377    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:13:43.128764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.128710    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"]
Oct 30 22:13:43.133126 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.133102    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"
Oct 30 22:13:43.186089 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.186050    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7a54db\""
Oct 30 22:13:43.212674 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.211326    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqt6t\" (UniqueName: \"kubernetes.io/projected/a77804c9-9480-43ac-83ac-c9f81d27728e-kube-api-access-pqt6t\") pod \"payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl\" (UID: \"a77804c9-9480-43ac-83ac-c9f81d27728e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"
Oct 30 22:13:43.212674 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.211396    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77804c9-9480-43ac-83ac-c9f81d27728e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl\" (UID: \"a77804c9-9480-43ac-83ac-c9f81d27728e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"
Oct 30 22:13:43.256333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.251334    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7a54db-dockercfg-lbphj\""
Oct 30 22:13:43.321159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.317195    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"]
Oct 30 22:13:43.321159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.318449    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqt6t\" (UniqueName: \"kubernetes.io/projected/a77804c9-9480-43ac-83ac-c9f81d27728e-kube-api-access-pqt6t\") pod \"payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl\" (UID: \"a77804c9-9480-43ac-83ac-c9f81d27728e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"
Oct 30 22:13:43.321159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.318504    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77804c9-9480-43ac-83ac-c9f81d27728e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl\" (UID: \"a77804c9-9480-43ac-83ac-c9f81d27728e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"
Oct 30 22:13:43.321159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.319873    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77804c9-9480-43ac-83ac-c9f81d27728e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl\" (UID: \"a77804c9-9480-43ac-83ac-c9f81d27728e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"
Oct 30 22:13:43.342155 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.333383    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"]
Oct 30 22:13:43.342550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.342522    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"
Oct 30 22:13:43.383711 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.383615    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"]
Oct 30 22:13:43.390717 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.389693    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqt6t\" (UniqueName: \"kubernetes.io/projected/a77804c9-9480-43ac-83ac-c9f81d27728e-kube-api-access-pqt6t\") pod \"payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl\" (UID: \"a77804c9-9480-43ac-83ac-c9f81d27728e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"
Oct 30 22:13:43.420430 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.420398    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhx4r\" (UniqueName: \"kubernetes.io/projected/edb147b9-1a46-40c0-965b-69d0f9645055-kube-api-access-dhx4r\") pod \"payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb\" (UID: \"edb147b9-1a46-40c0-965b-69d0f9645055\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"
Oct 30 22:13:43.420430 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.420433    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edb147b9-1a46-40c0-965b-69d0f9645055-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb\" (UID: \"edb147b9-1a46-40c0-965b-69d0f9645055\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"
Oct 30 22:13:43.448328 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.448284    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"
Oct 30 22:13:43.529844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.529797    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhx4r\" (UniqueName: \"kubernetes.io/projected/edb147b9-1a46-40c0-965b-69d0f9645055-kube-api-access-dhx4r\") pod \"payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb\" (UID: \"edb147b9-1a46-40c0-965b-69d0f9645055\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"
Oct 30 22:13:43.530022 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.529853    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edb147b9-1a46-40c0-965b-69d0f9645055-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb\" (UID: \"edb147b9-1a46-40c0-965b-69d0f9645055\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"
Oct 30 22:13:43.531031 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.531002    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edb147b9-1a46-40c0-965b-69d0f9645055-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb\" (UID: \"edb147b9-1a46-40c0-965b-69d0f9645055\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"
Oct 30 22:13:43.588202 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.588170    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhx4r\" (UniqueName: \"kubernetes.io/projected/edb147b9-1a46-40c0-965b-69d0f9645055-kube-api-access-dhx4r\") pod \"payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb\" (UID: \"edb147b9-1a46-40c0-965b-69d0f9645055\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"
Oct 30 22:13:43.693262 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.693235    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"
Oct 30 22:13:43.855871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:43.855844    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"]
Oct 30 22:13:43.856288 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:13:43.856264    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77804c9_9480_43ac_83ac_c9f81d27728e.slice/crio-ce157ae0cef5c1210673bd936e70e001b7637a6545cbda9d5d8c4d5eff725815 WatchSource:0}: Error finding container ce157ae0cef5c1210673bd936e70e001b7637a6545cbda9d5d8c4d5eff725815: Status 404 returned error can't find the container with id ce157ae0cef5c1210673bd936e70e001b7637a6545cbda9d5d8c4d5eff725815
Oct 30 22:13:44.079375 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:44.079346    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"]
Oct 30 22:13:44.080033 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:13:44.080007    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb147b9_1a46_40c0_965b_69d0f9645055.slice/crio-90d3e9f52321ef4bf15575762f72cf8ab3d09e1b8a5428a143097c316071bc59 WatchSource:0}: Error finding container 90d3e9f52321ef4bf15575762f72cf8ab3d09e1b8a5428a143097c316071bc59: Status 404 returned error can't find the container with id 90d3e9f52321ef4bf15575762f72cf8ab3d09e1b8a5428a143097c316071bc59
Oct 30 22:13:44.178128 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:44.178091    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb" event={"ID":"edb147b9-1a46-40c0-965b-69d0f9645055","Type":"ContainerStarted","Data":"0ef843bcf814301e8af4f177f0467c71976888ec48dbf801a6d16a634133a546"}
Oct 30 22:13:44.178128 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:44.178166    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb" event={"ID":"edb147b9-1a46-40c0-965b-69d0f9645055","Type":"ContainerStarted","Data":"90d3e9f52321ef4bf15575762f72cf8ab3d09e1b8a5428a143097c316071bc59"}
Oct 30 22:13:44.178672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:44.178197    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"
Oct 30 22:13:44.179843 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:44.179822    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl" event={"ID":"a77804c9-9480-43ac-83ac-c9f81d27728e","Type":"ContainerStarted","Data":"66be530f5d89028ddb7787f10155fdf62fd858b8a0a6f9db7ce9fe7469bf2e20"}
Oct 30 22:13:44.179954 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:44.179870    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl" event={"ID":"a77804c9-9480-43ac-83ac-c9f81d27728e","Type":"ContainerStarted","Data":"ce157ae0cef5c1210673bd936e70e001b7637a6545cbda9d5d8c4d5eff725815"}
Oct 30 22:13:44.179954 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:44.179893    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"
Oct 30 22:13:44.181428 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:44.181409    4553 generic.go:356] "Generic (PLEG): container finished" podID="f827754d-0840-4429-9ce1-8d63d32e6871" containerID="619c55ba101ea597fe9d68698c30cb10e208dadf3aa152d3b398174fecef4b96" exitCode=0
Oct 30 22:13:44.181514 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:44.181481    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc" event={"ID":"f827754d-0840-4429-9ce1-8d63d32e6871","Type":"ContainerDied","Data":"619c55ba101ea597fe9d68698c30cb10e208dadf3aa152d3b398174fecef4b96"}
Oct 30 22:13:44.181806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:44.181793    4553 scope.go:122] "RemoveContainer" containerID="619c55ba101ea597fe9d68698c30cb10e208dadf3aa152d3b398174fecef4b96"
Oct 30 22:13:44.183095 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:44.183064    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:44.183203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:44.183163    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:44.183259 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:44.183209    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:44.359160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:44.355866    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb" podStartSLOduration=1.355845171 podStartE2EDuration="1.355845171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:44.307705417 +0000 UTC m=+1875.042068452" watchObservedRunningTime="2026-10-30 22:13:44.355845171 +0000 UTC m=+1875.090208203"
Oct 30 22:13:44.607525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:44.607471    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl" podStartSLOduration=1.607456673 podStartE2EDuration="1.607456673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:44.606004083 +0000 UTC m=+1875.340367129" watchObservedRunningTime="2026-10-30 22:13:44.607456673 +0000 UTC m=+1875.341819753"
Oct 30 22:13:45.190147 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:45.190099    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc" event={"ID":"f827754d-0840-4429-9ce1-8d63d32e6871","Type":"ContainerStarted","Data":"8278e2943d86346bb1a7356aade97e2b63acc8f16ac161154fc343dd88670945"}
Oct 30 22:13:45.338526 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:45.338489    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:13:45.338802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:45.338741    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:13:45.338872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:45.338838    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc" podUID="f827754d-0840-4429-9ce1-8d63d32e6871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 22:13:46.339125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:46.338705    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:13:46.339125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:46.338768    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc" podUID="f827754d-0840-4429-9ce1-8d63d32e6871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 22:13:47.342550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:47.342510    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:13:47.343038 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:47.342575    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc" podUID="f827754d-0840-4429-9ce1-8d63d32e6871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 22:13:48.338731 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:48.338691    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:13:48.338943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:48.338748    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc" podUID="f827754d-0840-4429-9ce1-8d63d32e6871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 22:13:49.339438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:49.338993    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:13:49.339438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:49.339056    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc" podUID="f827754d-0840-4429-9ce1-8d63d32e6871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 22:13:50.342164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:50.341628    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Oct 30 22:13:50.342164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:50.341694    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc" podUID="f827754d-0840-4429-9ce1-8d63d32e6871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Oct 30 22:13:50.352327 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:50.352289    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:13:51.343691 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:51.343659    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:13:51.345012 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:51.344988    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:13:54.868198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:54.868167    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"]
Oct 30 22:13:54.868614 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:54.868371    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z" podUID="ba061980-6125-4a2c-9cc8-15dfa898d08c" containerName="maas-api" containerID="cri-o://ec95c581ccbb5f1883d3629c6470d4b00f96254490d10e7adac1e7666477c511" gracePeriod=30
Oct 30 22:13:55.187016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.186991    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:13:55.193623 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.193602    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"
Oct 30 22:13:55.193976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.193956    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"
Oct 30 22:13:55.278159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.272452    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba061980-6125-4a2c-9cc8-15dfa898d08c-kube-api-access-469sd\" (UniqueName: \"kubernetes.io/projected/ba061980-6125-4a2c-9cc8-15dfa898d08c-kube-api-access-469sd\") pod \"ba061980-6125-4a2c-9cc8-15dfa898d08c\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") "
Oct 30 22:13:55.278159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.272525    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba061980-6125-4a2c-9cc8-15dfa898d08c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba061980-6125-4a2c-9cc8-15dfa898d08c-openshift-service-ca\") pod \"ba061980-6125-4a2c-9cc8-15dfa898d08c\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") "
Oct 30 22:13:55.278159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.272574    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-maas-api-tls\") pod \"ba061980-6125-4a2c-9cc8-15dfa898d08c\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") "
Oct 30 22:13:55.278159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.272712    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-metrics-certs\") pod \"ba061980-6125-4a2c-9cc8-15dfa898d08c\" (UID: \"ba061980-6125-4a2c-9cc8-15dfa898d08c\") "
Oct 30 22:13:55.289750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.289708    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba061980-6125-4a2c-9cc8-15dfa898d08c-openshift-service-ca" pod "ba061980-6125-4a2c-9cc8-15dfa898d08c" (UID: "ba061980-6125-4a2c-9cc8-15dfa898d08c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:55.291461 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.291425    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba061980-6125-4a2c-9cc8-15dfa898d08c-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:55.293443 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.293417    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba061980-6125-4a2c-9cc8-15dfa898d08c-kube-api-access-469sd" pod "ba061980-6125-4a2c-9cc8-15dfa898d08c" (UID: "ba061980-6125-4a2c-9cc8-15dfa898d08c"). InnerVolumeSpecName "kube-api-access-469sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:55.297784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.297733    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-maas-api-tls" pod "ba061980-6125-4a2c-9cc8-15dfa898d08c" (UID: "ba061980-6125-4a2c-9cc8-15dfa898d08c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:55.298286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.298264    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-metrics-certs" pod "ba061980-6125-4a2c-9cc8-15dfa898d08c" (UID: "ba061980-6125-4a2c-9cc8-15dfa898d08c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:55.374933 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.374899    4553 generic.go:356] "Generic (PLEG): container finished" podID="ba061980-6125-4a2c-9cc8-15dfa898d08c" containerID="ec95c581ccbb5f1883d3629c6470d4b00f96254490d10e7adac1e7666477c511" exitCode=0
Oct 30 22:13:55.375233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.374977    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"
Oct 30 22:13:55.375374 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.374967    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z" event={"ID":"ba061980-6125-4a2c-9cc8-15dfa898d08c","Type":"ContainerDied","Data":"ec95c581ccbb5f1883d3629c6470d4b00f96254490d10e7adac1e7666477c511"}
Oct 30 22:13:55.375449 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.375371    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z" event={"ID":"ba061980-6125-4a2c-9cc8-15dfa898d08c","Type":"ContainerDied","Data":"1e7176e2a54cea4d598224171861afd8ec73969c8298e569e41891e32896406d"}
Oct 30 22:13:55.375449 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.375407    4553 scope.go:122] "RemoveContainer" containerID="ec95c581ccbb5f1883d3629c6470d4b00f96254490d10e7adac1e7666477c511"
Oct 30 22:13:55.389879 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.389861    4553 scope.go:122] "RemoveContainer" containerID="ec95c581ccbb5f1883d3629c6470d4b00f96254490d10e7adac1e7666477c511"
Oct 30 22:13:55.390247 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:13:55.390216    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec95c581ccbb5f1883d3629c6470d4b00f96254490d10e7adac1e7666477c511\": container with ID starting with ec95c581ccbb5f1883d3629c6470d4b00f96254490d10e7adac1e7666477c511 not found: ID does not exist" containerID="ec95c581ccbb5f1883d3629c6470d4b00f96254490d10e7adac1e7666477c511"
Oct 30 22:13:55.390343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.390257    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec95c581ccbb5f1883d3629c6470d4b00f96254490d10e7adac1e7666477c511"} err="failed to get container status \"ec95c581ccbb5f1883d3629c6470d4b00f96254490d10e7adac1e7666477c511\": rpc error: code = NotFound desc = could not find container \"ec95c581ccbb5f1883d3629c6470d4b00f96254490d10e7adac1e7666477c511\": container with ID starting with ec95c581ccbb5f1883d3629c6470d4b00f96254490d10e7adac1e7666477c511 not found: ID does not exist"
Oct 30 22:13:55.395124 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.394341    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-469sd\" (UniqueName: \"kubernetes.io/projected/ba061980-6125-4a2c-9cc8-15dfa898d08c-kube-api-access-469sd\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:55.395124 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.394380    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:55.395124 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.394394    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba061980-6125-4a2c-9cc8-15dfa898d08c-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:55.670240 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.670203    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"]
Oct 30 22:13:55.674101 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:55.674075    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7fabe91d-795b678cb8-fcv2z"]
Oct 30 22:13:56.326915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.326879    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"]
Oct 30 22:13:56.327586 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.327570    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ba061980-6125-4a2c-9cc8-15dfa898d08c" containerName="maas-api"
Oct 30 22:13:56.327630 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.327588    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba061980-6125-4a2c-9cc8-15dfa898d08c" containerName="maas-api"
Oct 30 22:13:56.327727 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.327717    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba061980-6125-4a2c-9cc8-15dfa898d08c" containerName="maas-api"
Oct 30 22:13:56.332585 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.332567    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"
Oct 30 22:13:56.388732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.386430    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-69a1d2\""
Oct 30 22:13:56.388732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.386484    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-69a1d2-dockercfg-9f9mq\""
Oct 30 22:13:56.388732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.388489    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"]
Oct 30 22:13:56.444959 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.428090    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189cad30-e568-4454-bf26-38fec456b0bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9\" (UID: \"189cad30-e568-4454-bf26-38fec456b0bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"
Oct 30 22:13:56.444959 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.428204    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltmw4\" (UniqueName: \"kubernetes.io/projected/189cad30-e568-4454-bf26-38fec456b0bd-kube-api-access-ltmw4\") pod \"payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9\" (UID: \"189cad30-e568-4454-bf26-38fec456b0bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"
Oct 30 22:13:56.538160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.538068    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189cad30-e568-4454-bf26-38fec456b0bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9\" (UID: \"189cad30-e568-4454-bf26-38fec456b0bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"
Oct 30 22:13:56.538342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.538177    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltmw4\" (UniqueName: \"kubernetes.io/projected/189cad30-e568-4454-bf26-38fec456b0bd-kube-api-access-ltmw4\") pod \"payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9\" (UID: \"189cad30-e568-4454-bf26-38fec456b0bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"
Oct 30 22:13:56.554206 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.539710    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189cad30-e568-4454-bf26-38fec456b0bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9\" (UID: \"189cad30-e568-4454-bf26-38fec456b0bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"
Oct 30 22:13:56.565222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.557499    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"]
Oct 30 22:13:56.565222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.562837    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"
Oct 30 22:13:56.570470 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.570421    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"]
Oct 30 22:13:56.605398 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.605356    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltmw4\" (UniqueName: \"kubernetes.io/projected/189cad30-e568-4454-bf26-38fec456b0bd-kube-api-access-ltmw4\") pod \"payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9\" (UID: \"189cad30-e568-4454-bf26-38fec456b0bd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"
Oct 30 22:13:56.639704 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.639673    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13383958-f736-4980-b167-e49792204928-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg\" (UID: \"13383958-f736-4980-b167-e49792204928\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"
Oct 30 22:13:56.639877 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.639816    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s2bh\" (UniqueName: \"kubernetes.io/projected/13383958-f736-4980-b167-e49792204928-kube-api-access-5s2bh\") pod \"payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg\" (UID: \"13383958-f736-4980-b167-e49792204928\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"
Oct 30 22:13:56.647707 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.647682    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"
Oct 30 22:13:56.740974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.740938    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13383958-f736-4980-b167-e49792204928-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg\" (UID: \"13383958-f736-4980-b167-e49792204928\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"
Oct 30 22:13:56.741173 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.741088    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s2bh\" (UniqueName: \"kubernetes.io/projected/13383958-f736-4980-b167-e49792204928-kube-api-access-5s2bh\") pod \"payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg\" (UID: \"13383958-f736-4980-b167-e49792204928\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"
Oct 30 22:13:56.741839 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.741813    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13383958-f736-4980-b167-e49792204928-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg\" (UID: \"13383958-f736-4980-b167-e49792204928\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"
Oct 30 22:13:56.809036 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.809003    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s2bh\" (UniqueName: \"kubernetes.io/projected/13383958-f736-4980-b167-e49792204928-kube-api-access-5s2bh\") pod \"payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg\" (UID: \"13383958-f736-4980-b167-e49792204928\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"
Oct 30 22:13:56.892412 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:56.892333    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"
Oct 30 22:13:57.036780 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:57.036751    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"]
Oct 30 22:13:57.038058 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:13:57.038032    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189cad30_e568_4454_bf26_38fec456b0bd.slice/crio-8741766a9ecdc1d5b7a3e419b31c1613ed44987ca62a67180a21a2baeed11127 WatchSource:0}: Error finding container 8741766a9ecdc1d5b7a3e419b31c1613ed44987ca62a67180a21a2baeed11127: Status 404 returned error can't find the container with id 8741766a9ecdc1d5b7a3e419b31c1613ed44987ca62a67180a21a2baeed11127
Oct 30 22:13:57.320374 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:57.320343    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"]
Oct 30 22:13:57.321285 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:13:57.321258    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13383958_f736_4980_b167_e49792204928.slice/crio-ff52f6a5d8f4e2c199e010044fc9fb628241b7a09256a9926f7b079ce2894439 WatchSource:0}: Error finding container ff52f6a5d8f4e2c199e010044fc9fb628241b7a09256a9926f7b079ce2894439: Status 404 returned error can't find the container with id ff52f6a5d8f4e2c199e010044fc9fb628241b7a09256a9926f7b079ce2894439
Oct 30 22:13:57.396321 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:57.396287    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba061980-6125-4a2c-9cc8-15dfa898d08c" path="/var/lib/kubelet/pods/ba061980-6125-4a2c-9cc8-15dfa898d08c/volumes"
Oct 30 22:13:57.396743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:57.396723    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg" event={"ID":"13383958-f736-4980-b167-e49792204928","Type":"ContainerStarted","Data":"ff52f6a5d8f4e2c199e010044fc9fb628241b7a09256a9926f7b079ce2894439"}
Oct 30 22:13:57.396803 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:57.396770    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9" event={"ID":"189cad30-e568-4454-bf26-38fec456b0bd","Type":"ContainerStarted","Data":"bc386620a693458f36853d91bbd15c2dd7714979768f3f10118da0ed7bc94a5f"}
Oct 30 22:13:57.396837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:57.396801    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"
Oct 30 22:13:57.396837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:57.396818    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9" event={"ID":"189cad30-e568-4454-bf26-38fec456b0bd","Type":"ContainerStarted","Data":"8741766a9ecdc1d5b7a3e419b31c1613ed44987ca62a67180a21a2baeed11127"}
Oct 30 22:13:57.513436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:57.513382    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9" podStartSLOduration=1.513367482 podStartE2EDuration="1.513367482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:57.511225529 +0000 UTC m=+1888.245588564" watchObservedRunningTime="2026-10-30 22:13:57.513367482 +0000 UTC m=+1888.247730512"
Oct 30 22:13:58.202181 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:13:58.202146    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:13:58.401896 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:58.401853    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg" event={"ID":"13383958-f736-4980-b167-e49792204928","Type":"ContainerStarted","Data":"469bf7ed253515fdd1509cd60534946a7df027631d97f5772438ea41976b648f"}
Oct 30 22:13:58.402361 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:58.401929    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"
Oct 30 22:13:58.521718 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:13:58.521614    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg" podStartSLOduration=2.521601671 podStartE2EDuration="2.521601671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:58.521394504 +0000 UTC m=+1889.255757536" watchObservedRunningTime="2026-10-30 22:13:58.521601671 +0000 UTC m=+1889.255964702"
Oct 30 22:14:08.405248 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:08.405209    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"
Oct 30 22:14:09.411883 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:09.411849    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"
Oct 30 22:14:21.590050 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:21.590010    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"]
Oct 30 22:14:21.700057 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:21.697260    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"]
Oct 30 22:14:26.590947 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:26.590906    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm" podUID="2d913198-413c-496e-b54f-a4823f53df45" containerName="payload-processing" containerID="cri-o://fb968ab314ed0693e95cad89e49c6e7c02c7eee45ea6eab88689818b14509c74" gracePeriod=25
Oct 30 22:14:26.698638 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:26.698599    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g" podUID="54fc71a0-ea97-4740-8f06-d41584d804dc" containerName="payload-pre-processing" containerID="cri-o://de97188224fae6b41f3121b2e988d5bd25a66c1a09d9761d21d7cd1cf78936f7" gracePeriod=25
Oct 30 22:14:26.890543 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:26.890516    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"
Oct 30 22:14:27.011660 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.011631    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"
Oct 30 22:14:27.014855 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.014831    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d913198-413c-496e-b54f-a4823f53df45-kube-api-access-gljjl\" (UniqueName: \"kubernetes.io/projected/2d913198-413c-496e-b54f-a4823f53df45-kube-api-access-gljjl\") pod \"2d913198-413c-496e-b54f-a4823f53df45\" (UID: \"2d913198-413c-496e-b54f-a4823f53df45\") "
Oct 30 22:14:27.014967 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.014914    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d913198-413c-496e-b54f-a4823f53df45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d913198-413c-496e-b54f-a4823f53df45-plugins-config-volume\") pod \"2d913198-413c-496e-b54f-a4823f53df45\" (UID: \"2d913198-413c-496e-b54f-a4823f53df45\") "
Oct 30 22:14:27.015292 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.015270    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d913198-413c-496e-b54f-a4823f53df45-plugins-config-volume" pod "2d913198-413c-496e-b54f-a4823f53df45" (UID: "2d913198-413c-496e-b54f-a4823f53df45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:27.017247 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.017222    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d913198-413c-496e-b54f-a4823f53df45-kube-api-access-gljjl" pod "2d913198-413c-496e-b54f-a4823f53df45" (UID: "2d913198-413c-496e-b54f-a4823f53df45"). InnerVolumeSpecName "kube-api-access-gljjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:27.116111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.116019    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fc71a0-ea97-4740-8f06-d41584d804dc-kube-api-access-ntcz5\" (UniqueName: \"kubernetes.io/projected/54fc71a0-ea97-4740-8f06-d41584d804dc-kube-api-access-ntcz5\") pod \"54fc71a0-ea97-4740-8f06-d41584d804dc\" (UID: \"54fc71a0-ea97-4740-8f06-d41584d804dc\") "
Oct 30 22:14:27.116111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.116093    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54fc71a0-ea97-4740-8f06-d41584d804dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54fc71a0-ea97-4740-8f06-d41584d804dc-plugins-config-volume\") pod \"54fc71a0-ea97-4740-8f06-d41584d804dc\" (UID: \"54fc71a0-ea97-4740-8f06-d41584d804dc\") "
Oct 30 22:14:27.116528 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.116506    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54fc71a0-ea97-4740-8f06-d41584d804dc-plugins-config-volume" pod "54fc71a0-ea97-4740-8f06-d41584d804dc" (UID: "54fc71a0-ea97-4740-8f06-d41584d804dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:27.116663 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.116647    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54fc71a0-ea97-4740-8f06-d41584d804dc-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:27.116703 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.116667    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d913198-413c-496e-b54f-a4823f53df45-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:27.116703 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.116676    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gljjl\" (UniqueName: \"kubernetes.io/projected/2d913198-413c-496e-b54f-a4823f53df45-kube-api-access-gljjl\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:27.118467 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.118448    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fc71a0-ea97-4740-8f06-d41584d804dc-kube-api-access-ntcz5" pod "54fc71a0-ea97-4740-8f06-d41584d804dc" (UID: "54fc71a0-ea97-4740-8f06-d41584d804dc"). InnerVolumeSpecName "kube-api-access-ntcz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:27.217954 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.217912    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntcz5\" (UniqueName: \"kubernetes.io/projected/54fc71a0-ea97-4740-8f06-d41584d804dc-kube-api-access-ntcz5\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:27.591360 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.591328    4553 generic.go:356] "Generic (PLEG): container finished" podID="54fc71a0-ea97-4740-8f06-d41584d804dc" containerID="de97188224fae6b41f3121b2e988d5bd25a66c1a09d9761d21d7cd1cf78936f7" exitCode=0
Oct 30 22:14:27.591802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.591385    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"
Oct 30 22:14:27.591802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.591389    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g" event={"ID":"54fc71a0-ea97-4740-8f06-d41584d804dc","Type":"ContainerDied","Data":"de97188224fae6b41f3121b2e988d5bd25a66c1a09d9761d21d7cd1cf78936f7"}
Oct 30 22:14:27.591802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.591443    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g" event={"ID":"54fc71a0-ea97-4740-8f06-d41584d804dc","Type":"ContainerDied","Data":"89b7491dfcef017847a7acd3b0bc0efc7a683de5cc2bf268066d6d6cc2ac6432"}
Oct 30 22:14:27.591802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.591469    4553 scope.go:122] "RemoveContainer" containerID="de97188224fae6b41f3121b2e988d5bd25a66c1a09d9761d21d7cd1cf78936f7"
Oct 30 22:14:27.593169 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.593123    4553 generic.go:356] "Generic (PLEG): container finished" podID="2d913198-413c-496e-b54f-a4823f53df45" containerID="fb968ab314ed0693e95cad89e49c6e7c02c7eee45ea6eab88689818b14509c74" exitCode=0
Oct 30 22:14:27.593290 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.593234    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"
Oct 30 22:14:27.593355 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.593233    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm" event={"ID":"2d913198-413c-496e-b54f-a4823f53df45","Type":"ContainerDied","Data":"fb968ab314ed0693e95cad89e49c6e7c02c7eee45ea6eab88689818b14509c74"}
Oct 30 22:14:27.593405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.593361    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm" event={"ID":"2d913198-413c-496e-b54f-a4823f53df45","Type":"ContainerDied","Data":"829e469e7f53dcf507d9a65b24e60068a03ffb8b13b9744cea10a494b2864bcb"}
Oct 30 22:14:27.602119 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.602096    4553 scope.go:122] "RemoveContainer" containerID="de97188224fae6b41f3121b2e988d5bd25a66c1a09d9761d21d7cd1cf78936f7"
Oct 30 22:14:27.602489 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:14:27.602464    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de97188224fae6b41f3121b2e988d5bd25a66c1a09d9761d21d7cd1cf78936f7\": container with ID starting with de97188224fae6b41f3121b2e988d5bd25a66c1a09d9761d21d7cd1cf78936f7 not found: ID does not exist" containerID="de97188224fae6b41f3121b2e988d5bd25a66c1a09d9761d21d7cd1cf78936f7"
Oct 30 22:14:27.602582 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.602502    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de97188224fae6b41f3121b2e988d5bd25a66c1a09d9761d21d7cd1cf78936f7"} err="failed to get container status \"de97188224fae6b41f3121b2e988d5bd25a66c1a09d9761d21d7cd1cf78936f7\": rpc error: code = NotFound desc = could not find container \"de97188224fae6b41f3121b2e988d5bd25a66c1a09d9761d21d7cd1cf78936f7\": container with ID starting with de97188224fae6b41f3121b2e988d5bd25a66c1a09d9761d21d7cd1cf78936f7 not found: ID does not exist"
Oct 30 22:14:27.602582 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.602524    4553 scope.go:122] "RemoveContainer" containerID="fb968ab314ed0693e95cad89e49c6e7c02c7eee45ea6eab88689818b14509c74"
Oct 30 22:14:27.614727 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.614700    4553 scope.go:122] "RemoveContainer" containerID="fb968ab314ed0693e95cad89e49c6e7c02c7eee45ea6eab88689818b14509c74"
Oct 30 22:14:27.615040 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:14:27.615021    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb968ab314ed0693e95cad89e49c6e7c02c7eee45ea6eab88689818b14509c74\": container with ID starting with fb968ab314ed0693e95cad89e49c6e7c02c7eee45ea6eab88689818b14509c74 not found: ID does not exist" containerID="fb968ab314ed0693e95cad89e49c6e7c02c7eee45ea6eab88689818b14509c74"
Oct 30 22:14:27.615091 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.615048    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb968ab314ed0693e95cad89e49c6e7c02c7eee45ea6eab88689818b14509c74"} err="failed to get container status \"fb968ab314ed0693e95cad89e49c6e7c02c7eee45ea6eab88689818b14509c74\": rpc error: code = NotFound desc = could not find container \"fb968ab314ed0693e95cad89e49c6e7c02c7eee45ea6eab88689818b14509c74\": container with ID starting with fb968ab314ed0693e95cad89e49c6e7c02c7eee45ea6eab88689818b14509c74 not found: ID does not exist"
Oct 30 22:14:27.758623 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.758540    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"]
Oct 30 22:14:27.763231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.763205    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-859078-7dbf99bcd9-pqdvm"]
Oct 30 22:14:27.872444 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.872415    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"]
Oct 30 22:14:27.875292 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:27.875267    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-65444c9ddc-lgc9g"]
Oct 30 22:14:28.203506 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:14:28.203471    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:14:29.393343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:29.393314    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d913198-413c-496e-b54f-a4823f53df45" path="/var/lib/kubelet/pods/2d913198-413c-496e-b54f-a4823f53df45/volumes"
Oct 30 22:14:29.393725 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:29.393703    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54fc71a0-ea97-4740-8f06-d41584d804dc" path="/var/lib/kubelet/pods/54fc71a0-ea97-4740-8f06-d41584d804dc/volumes"
Oct 30 22:14:47.046449 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:47.046395    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"]
Oct 30 22:14:47.047026 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:47.046645    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg" podUID="7a268712-b551-4bdd-8fbd-70e124508a44" containerName="istio-proxy" containerID="cri-o://0066d0c30e041d40f3e01d85ebb13ac339ef9a598c461ce89f91bee141afaa93" gracePeriod=30
Oct 30 22:14:47.051241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:47.051217    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"]
Oct 30 22:14:47.051412 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:47.051392    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm" podUID="6f396aaf-e41b-49c4-8734-cd729fa95810" containerName="istio-proxy" containerID="cri-o://16b86f83287a350cde6c678e9ed71beebe709a039a7ba233af6fcb76b5ef282a" gracePeriod=30
Oct 30 22:14:47.082128 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:47.082090    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"]
Oct 30 22:14:47.082337 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:47.082316    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv" podUID="a765ef62-b023-42d6-96c1-8e7715f11f3c" containerName="istio-proxy" containerID="cri-o://6ef6f7b7651322d793e73addc4d47914d03c8ca947e9ae38d6f158fcd82be4f9" gracePeriod=30
Oct 30 22:14:47.154371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:47.154212    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:14:47.154371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:47.154300    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv" podUID="a765ef62-b023-42d6-96c1-8e7715f11f3c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:14:47.875542 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:47.875491    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:14:47.875731 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:47.875611    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm" podUID="6f396aaf-e41b-49c4-8734-cd729fa95810" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:14:47.931311 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:47.931259    4553 patch_prober.go:29] interesting pod/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:14:47.931486 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:47.931336    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg" podUID="7a268712-b551-4bdd-8fbd-70e124508a44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:14:48.469342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.469309    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"]
Oct 30 22:14:48.469794 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.469538    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv" podUID="f72b872a-65b7-4923-b5f9-fbb01be82e41" containerName="maas-api" containerID="cri-o://aee38380a5d5d5227fa5d405651d0b1d73f9aef6732031aafc4cfd8f6c7dfb30" gracePeriod=30
Oct 30 22:14:48.811057 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.811030    4553 generic.go:356] "Generic (PLEG): container finished" podID="f72b872a-65b7-4923-b5f9-fbb01be82e41" containerID="aee38380a5d5d5227fa5d405651d0b1d73f9aef6732031aafc4cfd8f6c7dfb30" exitCode=0
Oct 30 22:14:48.811242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.811100    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv" event={"ID":"f72b872a-65b7-4923-b5f9-fbb01be82e41","Type":"ContainerDied","Data":"aee38380a5d5d5227fa5d405651d0b1d73f9aef6732031aafc4cfd8f6c7dfb30"}
Oct 30 22:14:48.811242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.811185    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv" event={"ID":"f72b872a-65b7-4923-b5f9-fbb01be82e41","Type":"ContainerDied","Data":"9dff6f7d176e5a3bf7992c2c04e86f18b985292f4d4921ae197137c860a09879"}
Oct 30 22:14:48.811242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.811216    4553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dff6f7d176e5a3bf7992c2c04e86f18b985292f4d4921ae197137c860a09879"
Oct 30 22:14:48.813122 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.813106    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:14:48.964978 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.964939    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-metrics-certs\") pod \"f72b872a-65b7-4923-b5f9-fbb01be82e41\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") "
Oct 30 22:14:48.965239 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.965219    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-maas-api-tls\") pod \"f72b872a-65b7-4923-b5f9-fbb01be82e41\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") "
Oct 30 22:14:48.965319 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.965292    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72b872a-65b7-4923-b5f9-fbb01be82e41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72b872a-65b7-4923-b5f9-fbb01be82e41-openshift-service-ca\") pod \"f72b872a-65b7-4923-b5f9-fbb01be82e41\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") "
Oct 30 22:14:48.965376 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.965323    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f72b872a-65b7-4923-b5f9-fbb01be82e41-kube-api-access-qmlc2\" (UniqueName: \"kubernetes.io/projected/f72b872a-65b7-4923-b5f9-fbb01be82e41-kube-api-access-qmlc2\") pod \"f72b872a-65b7-4923-b5f9-fbb01be82e41\" (UID: \"f72b872a-65b7-4923-b5f9-fbb01be82e41\") "
Oct 30 22:14:48.965730 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.965699    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72b872a-65b7-4923-b5f9-fbb01be82e41-openshift-service-ca" pod "f72b872a-65b7-4923-b5f9-fbb01be82e41" (UID: "f72b872a-65b7-4923-b5f9-fbb01be82e41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:48.965976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.965954    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f72b872a-65b7-4923-b5f9-fbb01be82e41-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:48.968452 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.968421    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-maas-api-tls" pod "f72b872a-65b7-4923-b5f9-fbb01be82e41" (UID: "f72b872a-65b7-4923-b5f9-fbb01be82e41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:48.969320 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.969295    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72b872a-65b7-4923-b5f9-fbb01be82e41-kube-api-access-qmlc2" pod "f72b872a-65b7-4923-b5f9-fbb01be82e41" (UID: "f72b872a-65b7-4923-b5f9-fbb01be82e41"). InnerVolumeSpecName "kube-api-access-qmlc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:48.969683 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:48.969659    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-metrics-certs" pod "f72b872a-65b7-4923-b5f9-fbb01be82e41" (UID: "f72b872a-65b7-4923-b5f9-fbb01be82e41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:49.092730 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:49.092627    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:49.092730 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:49.092676    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f72b872a-65b7-4923-b5f9-fbb01be82e41-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:49.092730 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:49.092690    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmlc2\" (UniqueName: \"kubernetes.io/projected/f72b872a-65b7-4923-b5f9-fbb01be82e41-kube-api-access-qmlc2\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:49.818479 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:49.818402    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"
Oct 30 22:14:49.986328 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:49.986292    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"]
Oct 30 22:14:50.035994 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:50.035958    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2bc5e3-7445b9cd75-dgcmv"]
Oct 30 22:14:51.394014 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:51.393983    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f72b872a-65b7-4923-b5f9-fbb01be82e41" path="/var/lib/kubelet/pods/f72b872a-65b7-4923-b5f9-fbb01be82e41/volumes"
Oct 30 22:14:51.841938 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:51.841907    4553 generic.go:356] "Generic (PLEG): container finished" podID="38824b8a-8c76-412c-9d99-eb8955f8d237" containerID="7b6b263643855f0afd3042400b79885698766cea56fed4bc4c929c90faee8d22" exitCode=0
Oct 30 22:14:51.841938 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:51.841929    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn" event={"ID":"38824b8a-8c76-412c-9d99-eb8955f8d237","Type":"ContainerDied","Data":"7b6b263643855f0afd3042400b79885698766cea56fed4bc4c929c90faee8d22"}
Oct 30 22:14:51.842497 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:51.842480    4553 scope.go:122] "RemoveContainer" containerID="7b6b263643855f0afd3042400b79885698766cea56fed4bc4c929c90faee8d22"
Oct 30 22:14:51.844395 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:51.844364    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:14:51.844519 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:51.844446    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:14:51.844519 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:51.844495    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:14:52.122580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.122552    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:14:52.373881 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.373855    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:14:52.447167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.439460    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-credential-socket\") pod \"7a268712-b551-4bdd-8fbd-70e124508a44\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") "
Oct 30 22:14:52.447167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.439588    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-workload-certs\") pod \"7a268712-b551-4bdd-8fbd-70e124508a44\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") "
Oct 30 22:14:52.447167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.439632    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a268712-b551-4bdd-8fbd-70e124508a44-istio-token\" (UniqueName: \"kubernetes.io/projected/7a268712-b551-4bdd-8fbd-70e124508a44-istio-token\") pod \"7a268712-b551-4bdd-8fbd-70e124508a44\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") "
Oct 30 22:14:52.447167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.439687    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a268712-b551-4bdd-8fbd-70e124508a44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a268712-b551-4bdd-8fbd-70e124508a44-istio-podinfo\") pod \"7a268712-b551-4bdd-8fbd-70e124508a44\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") "
Oct 30 22:14:52.447167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.439756    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-workload-socket\") pod \"7a268712-b551-4bdd-8fbd-70e124508a44\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") "
Oct 30 22:14:52.447167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.439826    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a268712-b551-4bdd-8fbd-70e124508a44-kube-api-access-g8n7h\" (UniqueName: \"kubernetes.io/projected/7a268712-b551-4bdd-8fbd-70e124508a44-kube-api-access-g8n7h\") pod \"7a268712-b551-4bdd-8fbd-70e124508a44\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") "
Oct 30 22:14:52.447167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.439868    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a268712-b551-4bdd-8fbd-70e124508a44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a268712-b551-4bdd-8fbd-70e124508a44-istiod-ca-cert\") pod \"7a268712-b551-4bdd-8fbd-70e124508a44\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") "
Oct 30 22:14:52.447167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.439895    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-istio-data\") pod \"7a268712-b551-4bdd-8fbd-70e124508a44\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") "
Oct 30 22:14:52.447167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.439937    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-istio-envoy\") pod \"7a268712-b551-4bdd-8fbd-70e124508a44\" (UID: \"7a268712-b551-4bdd-8fbd-70e124508a44\") "
Oct 30 22:14:52.451605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.450661    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-workload-socket" pod "7a268712-b551-4bdd-8fbd-70e124508a44" (UID: "7a268712-b551-4bdd-8fbd-70e124508a44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.451605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.450764    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-credential-socket" pod "7a268712-b551-4bdd-8fbd-70e124508a44" (UID: "7a268712-b551-4bdd-8fbd-70e124508a44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.451605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.450816    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-istio-data" pod "7a268712-b551-4bdd-8fbd-70e124508a44" (UID: "7a268712-b551-4bdd-8fbd-70e124508a44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.451605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.450865    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-workload-certs" pod "7a268712-b551-4bdd-8fbd-70e124508a44" (UID: "7a268712-b551-4bdd-8fbd-70e124508a44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.451605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.451012    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a268712-b551-4bdd-8fbd-70e124508a44-istiod-ca-cert" pod "7a268712-b551-4bdd-8fbd-70e124508a44" (UID: "7a268712-b551-4bdd-8fbd-70e124508a44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:52.451605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.451479    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:14:52.453598 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.453565    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-istio-envoy" pod "7a268712-b551-4bdd-8fbd-70e124508a44" (UID: "7a268712-b551-4bdd-8fbd-70e124508a44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.454201 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.454165    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a268712-b551-4bdd-8fbd-70e124508a44-istio-podinfo" pod "7a268712-b551-4bdd-8fbd-70e124508a44" (UID: "7a268712-b551-4bdd-8fbd-70e124508a44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:52.455906 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.455865    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a268712-b551-4bdd-8fbd-70e124508a44-kube-api-access-g8n7h" pod "7a268712-b551-4bdd-8fbd-70e124508a44" (UID: "7a268712-b551-4bdd-8fbd-70e124508a44"). InnerVolumeSpecName "kube-api-access-g8n7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:52.456008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.455943    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a268712-b551-4bdd-8fbd-70e124508a44-istio-token" pod "7a268712-b551-4bdd-8fbd-70e124508a44" (UID: "7a268712-b551-4bdd-8fbd-70e124508a44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:52.470830 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.470803    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:14:52.541700 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.541666    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-token\" (UniqueName: \"kubernetes.io/projected/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-token\") pod \"6f396aaf-e41b-49c4-8734-cd729fa95810\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") "
Oct 30 22:14:52.542090 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.542071    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-workload-certs\") pod \"6f396aaf-e41b-49c4-8734-cd729fa95810\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") "
Oct 30 22:14:52.542446 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.542397    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-workload-certs" pod "6f396aaf-e41b-49c4-8734-cd729fa95810" (UID: "6f396aaf-e41b-49c4-8734-cd729fa95810"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.542572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.542486    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-data\") pod \"a765ef62-b023-42d6-96c1-8e7715f11f3c\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") "
Oct 30 22:14:52.542572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.542508    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-data\") pod \"6f396aaf-e41b-49c4-8734-cd729fa95810\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") "
Oct 30 22:14:52.542751 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.542717    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-data" pod "6f396aaf-e41b-49c4-8734-cd729fa95810" (UID: "6f396aaf-e41b-49c4-8734-cd729fa95810"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.542899 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.542808    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-data" pod "a765ef62-b023-42d6-96c1-8e7715f11f3c" (UID: "a765ef62-b023-42d6-96c1-8e7715f11f3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.542899 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.542844    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-token\" (UniqueName: \"kubernetes.io/projected/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-token\") pod \"a765ef62-b023-42d6-96c1-8e7715f11f3c\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") "
Oct 30 22:14:52.542899 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.542875    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f396aaf-e41b-49c4-8734-cd729fa95810-kube-api-access-zwp84\" (UniqueName: \"kubernetes.io/projected/6f396aaf-e41b-49c4-8734-cd729fa95810-kube-api-access-zwp84\") pod \"6f396aaf-e41b-49c4-8734-cd729fa95810\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") "
Oct 30 22:14:52.544237 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.544211    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-podinfo\") pod \"a765ef62-b023-42d6-96c1-8e7715f11f3c\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") "
Oct 30 22:14:52.544343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.544262    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-envoy\") pod \"6f396aaf-e41b-49c4-8734-cd729fa95810\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") "
Oct 30 22:14:52.544343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.544310    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-workload-certs\") pod \"a765ef62-b023-42d6-96c1-8e7715f11f3c\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") "
Oct 30 22:14:52.544475 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.544456    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a765ef62-b023-42d6-96c1-8e7715f11f3c-kube-api-access-nz5nh\" (UniqueName: \"kubernetes.io/projected/a765ef62-b023-42d6-96c1-8e7715f11f3c-kube-api-access-nz5nh\") pod \"a765ef62-b023-42d6-96c1-8e7715f11f3c\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") "
Oct 30 22:14:52.544554 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.544538    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-workload-socket\") pod \"6f396aaf-e41b-49c4-8734-cd729fa95810\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") "
Oct 30 22:14:52.544619 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.544578    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-credential-socket\") pod \"a765ef62-b023-42d6-96c1-8e7715f11f3c\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") "
Oct 30 22:14:52.544619 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.544612    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-envoy\") pod \"a765ef62-b023-42d6-96c1-8e7715f11f3c\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") "
Oct 30 22:14:52.544719 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.544692    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-workload-socket\") pod \"a765ef62-b023-42d6-96c1-8e7715f11f3c\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") "
Oct 30 22:14:52.544787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.544770    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-credential-socket\") pod \"6f396aaf-e41b-49c4-8734-cd729fa95810\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") "
Oct 30 22:14:52.544844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.544794    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-token" pod "6f396aaf-e41b-49c4-8734-cd729fa95810" (UID: "6f396aaf-e41b-49c4-8734-cd729fa95810"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:52.544844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.544810    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f396aaf-e41b-49c4-8734-cd729fa95810-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f396aaf-e41b-49c4-8734-cd729fa95810-istiod-ca-cert\") pod \"6f396aaf-e41b-49c4-8734-cd729fa95810\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") "
Oct 30 22:14:52.545089 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.544847    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-podinfo\") pod \"6f396aaf-e41b-49c4-8734-cd729fa95810\" (UID: \"6f396aaf-e41b-49c4-8734-cd729fa95810\") "
Oct 30 22:14:52.545089 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.544888    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a765ef62-b023-42d6-96c1-8e7715f11f3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a765ef62-b023-42d6-96c1-8e7715f11f3c-istiod-ca-cert\") pod \"a765ef62-b023-42d6-96c1-8e7715f11f3c\" (UID: \"a765ef62-b023-42d6-96c1-8e7715f11f3c\") "
Oct 30 22:14:52.545272 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.545109    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-credential-socket" pod "a765ef62-b023-42d6-96c1-8e7715f11f3c" (UID: "a765ef62-b023-42d6-96c1-8e7715f11f3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.545497 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.545474    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a765ef62-b023-42d6-96c1-8e7715f11f3c-istiod-ca-cert" pod "a765ef62-b023-42d6-96c1-8e7715f11f3c" (UID: "a765ef62-b023-42d6-96c1-8e7715f11f3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:52.545768 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.545747    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f396aaf-e41b-49c4-8734-cd729fa95810-istiod-ca-cert" pod "6f396aaf-e41b-49c4-8734-cd729fa95810" (UID: "6f396aaf-e41b-49c4-8734-cd729fa95810"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:52.545954 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.545928    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-credential-socket" pod "6f396aaf-e41b-49c4-8734-cd729fa95810" (UID: "6f396aaf-e41b-49c4-8734-cd729fa95810"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.546171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546128    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a268712-b551-4bdd-8fbd-70e124508a44-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546175    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546190    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f396aaf-e41b-49c4-8734-cd729fa95810-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546202    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546211    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a765ef62-b023-42d6-96c1-8e7715f11f3c-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546220    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546229    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546238    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8n7h\" (UniqueName: \"kubernetes.io/projected/7a268712-b551-4bdd-8fbd-70e124508a44-kube-api-access-g8n7h\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546248    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a268712-b551-4bdd-8fbd-70e124508a44-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546257    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546267    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546275    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546283    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546291    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546299    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a268712-b551-4bdd-8fbd-70e124508a44-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546307    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546315    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a268712-b551-4bdd-8fbd-70e124508a44-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.546732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546453    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-workload-certs" pod "a765ef62-b023-42d6-96c1-8e7715f11f3c" (UID: "a765ef62-b023-42d6-96c1-8e7715f11f3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.547149 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546803    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-envoy" pod "6f396aaf-e41b-49c4-8734-cd729fa95810" (UID: "6f396aaf-e41b-49c4-8734-cd729fa95810"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.547149 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.546994    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-workload-socket" pod "6f396aaf-e41b-49c4-8734-cd729fa95810" (UID: "6f396aaf-e41b-49c4-8734-cd729fa95810"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.547149 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.547061    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-podinfo" pod "a765ef62-b023-42d6-96c1-8e7715f11f3c" (UID: "a765ef62-b023-42d6-96c1-8e7715f11f3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:52.547329 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.547263    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-workload-socket" pod "a765ef62-b023-42d6-96c1-8e7715f11f3c" (UID: "a765ef62-b023-42d6-96c1-8e7715f11f3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.548613 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.548557    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f396aaf-e41b-49c4-8734-cd729fa95810-kube-api-access-zwp84" pod "6f396aaf-e41b-49c4-8734-cd729fa95810" (UID: "6f396aaf-e41b-49c4-8734-cd729fa95810"). InnerVolumeSpecName "kube-api-access-zwp84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:52.548720 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.548657    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-token" pod "a765ef62-b023-42d6-96c1-8e7715f11f3c" (UID: "a765ef62-b023-42d6-96c1-8e7715f11f3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:52.548720 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.548671    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-podinfo" pod "6f396aaf-e41b-49c4-8734-cd729fa95810" (UID: "6f396aaf-e41b-49c4-8734-cd729fa95810"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:14:52.549244 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.549220    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a765ef62-b023-42d6-96c1-8e7715f11f3c-kube-api-access-nz5nh" pod "a765ef62-b023-42d6-96c1-8e7715f11f3c" (UID: "a765ef62-b023-42d6-96c1-8e7715f11f3c"). InnerVolumeSpecName "kube-api-access-nz5nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:52.550662 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.550633    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-envoy" pod "a765ef62-b023-42d6-96c1-8e7715f11f3c" (UID: "a765ef62-b023-42d6-96c1-8e7715f11f3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:52.648023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.647943    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.648023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.647980    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.648023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.647995    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.648023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.648008    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwp84\" (UniqueName: \"kubernetes.io/projected/6f396aaf-e41b-49c4-8734-cd729fa95810-kube-api-access-zwp84\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.648023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.648017    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.648359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.648027    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.648359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.648036    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.648359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.648045    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz5nh\" (UniqueName: \"kubernetes.io/projected/a765ef62-b023-42d6-96c1-8e7715f11f3c-kube-api-access-nz5nh\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.648359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.648052    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f396aaf-e41b-49c4-8734-cd729fa95810-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.648359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.648060    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a765ef62-b023-42d6-96c1-8e7715f11f3c-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:52.853649 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.853607    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn" event={"ID":"38824b8a-8c76-412c-9d99-eb8955f8d237","Type":"ContainerStarted","Data":"6969faaf03f1d297906e16cd281e15ce4db5ba823baa32afd20deeb28d2c5ab8"}
Oct 30 22:14:52.855091 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.855068    4553 generic.go:356] "Generic (PLEG): container finished" podID="a765ef62-b023-42d6-96c1-8e7715f11f3c" containerID="6ef6f7b7651322d793e73addc4d47914d03c8ca947e9ae38d6f158fcd82be4f9" exitCode=0
Oct 30 22:14:52.855223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.855109    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"
Oct 30 22:14:52.855223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.855165    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv" event={"ID":"a765ef62-b023-42d6-96c1-8e7715f11f3c","Type":"ContainerDied","Data":"6ef6f7b7651322d793e73addc4d47914d03c8ca947e9ae38d6f158fcd82be4f9"}
Oct 30 22:14:52.855342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.855236    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv" event={"ID":"a765ef62-b023-42d6-96c1-8e7715f11f3c","Type":"ContainerDied","Data":"61ccc6c7244def3e4c71bc4998b6b73dc741a7e6d93f5420009c3457300e8eb7"}
Oct 30 22:14:52.855342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.855270    4553 scope.go:122] "RemoveContainer" containerID="6ef6f7b7651322d793e73addc4d47914d03c8ca947e9ae38d6f158fcd82be4f9"
Oct 30 22:14:52.856895 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.856713    4553 generic.go:356] "Generic (PLEG): container finished" podID="7a268712-b551-4bdd-8fbd-70e124508a44" containerID="0066d0c30e041d40f3e01d85ebb13ac339ef9a598c461ce89f91bee141afaa93" exitCode=0
Oct 30 22:14:52.856895 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.856755    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"
Oct 30 22:14:52.856895 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.856795    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg" event={"ID":"7a268712-b551-4bdd-8fbd-70e124508a44","Type":"ContainerDied","Data":"0066d0c30e041d40f3e01d85ebb13ac339ef9a598c461ce89f91bee141afaa93"}
Oct 30 22:14:52.856895 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.856845    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg" event={"ID":"7a268712-b551-4bdd-8fbd-70e124508a44","Type":"ContainerDied","Data":"99ef99ea82340ff1da68b0517adc1c1e7940167aeb0166988351b0b99b9ddcd9"}
Oct 30 22:14:52.859390 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.859369    4553 generic.go:356] "Generic (PLEG): container finished" podID="6f396aaf-e41b-49c4-8734-cd729fa95810" containerID="16b86f83287a350cde6c678e9ed71beebe709a039a7ba233af6fcb76b5ef282a" exitCode=0
Oct 30 22:14:52.859515 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.859403    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm" event={"ID":"6f396aaf-e41b-49c4-8734-cd729fa95810","Type":"ContainerDied","Data":"16b86f83287a350cde6c678e9ed71beebe709a039a7ba233af6fcb76b5ef282a"}
Oct 30 22:14:52.859515 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.859424    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"
Oct 30 22:14:52.859515 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.859462    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm" event={"ID":"6f396aaf-e41b-49c4-8734-cd729fa95810","Type":"ContainerDied","Data":"2c705d9fae57b617483104a3ac90094ffe29579e841ae98acb608864a996cde1"}
Oct 30 22:14:52.867107 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.867091    4553 scope.go:122] "RemoveContainer" containerID="6ef6f7b7651322d793e73addc4d47914d03c8ca947e9ae38d6f158fcd82be4f9"
Oct 30 22:14:52.867453 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:14:52.867432    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef6f7b7651322d793e73addc4d47914d03c8ca947e9ae38d6f158fcd82be4f9\": container with ID starting with 6ef6f7b7651322d793e73addc4d47914d03c8ca947e9ae38d6f158fcd82be4f9 not found: ID does not exist" containerID="6ef6f7b7651322d793e73addc4d47914d03c8ca947e9ae38d6f158fcd82be4f9"
Oct 30 22:14:52.867544 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.867460    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef6f7b7651322d793e73addc4d47914d03c8ca947e9ae38d6f158fcd82be4f9"} err="failed to get container status \"6ef6f7b7651322d793e73addc4d47914d03c8ca947e9ae38d6f158fcd82be4f9\": rpc error: code = NotFound desc = could not find container \"6ef6f7b7651322d793e73addc4d47914d03c8ca947e9ae38d6f158fcd82be4f9\": container with ID starting with 6ef6f7b7651322d793e73addc4d47914d03c8ca947e9ae38d6f158fcd82be4f9 not found: ID does not exist"
Oct 30 22:14:52.867544 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.867477    4553 scope.go:122] "RemoveContainer" containerID="0066d0c30e041d40f3e01d85ebb13ac339ef9a598c461ce89f91bee141afaa93"
Oct 30 22:14:52.877416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.877396    4553 scope.go:122] "RemoveContainer" containerID="0066d0c30e041d40f3e01d85ebb13ac339ef9a598c461ce89f91bee141afaa93"
Oct 30 22:14:52.877716 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:14:52.877697    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0066d0c30e041d40f3e01d85ebb13ac339ef9a598c461ce89f91bee141afaa93\": container with ID starting with 0066d0c30e041d40f3e01d85ebb13ac339ef9a598c461ce89f91bee141afaa93 not found: ID does not exist" containerID="0066d0c30e041d40f3e01d85ebb13ac339ef9a598c461ce89f91bee141afaa93"
Oct 30 22:14:52.877769 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.877724    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0066d0c30e041d40f3e01d85ebb13ac339ef9a598c461ce89f91bee141afaa93"} err="failed to get container status \"0066d0c30e041d40f3e01d85ebb13ac339ef9a598c461ce89f91bee141afaa93\": rpc error: code = NotFound desc = could not find container \"0066d0c30e041d40f3e01d85ebb13ac339ef9a598c461ce89f91bee141afaa93\": container with ID starting with 0066d0c30e041d40f3e01d85ebb13ac339ef9a598c461ce89f91bee141afaa93 not found: ID does not exist"
Oct 30 22:14:52.877769 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.877741    4553 scope.go:122] "RemoveContainer" containerID="16b86f83287a350cde6c678e9ed71beebe709a039a7ba233af6fcb76b5ef282a"
Oct 30 22:14:52.887079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.887059    4553 scope.go:122] "RemoveContainer" containerID="16b86f83287a350cde6c678e9ed71beebe709a039a7ba233af6fcb76b5ef282a"
Oct 30 22:14:52.887412 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:14:52.887390    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b86f83287a350cde6c678e9ed71beebe709a039a7ba233af6fcb76b5ef282a\": container with ID starting with 16b86f83287a350cde6c678e9ed71beebe709a039a7ba233af6fcb76b5ef282a not found: ID does not exist" containerID="16b86f83287a350cde6c678e9ed71beebe709a039a7ba233af6fcb76b5ef282a"
Oct 30 22:14:52.887519 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:52.887419    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b86f83287a350cde6c678e9ed71beebe709a039a7ba233af6fcb76b5ef282a"} err="failed to get container status \"16b86f83287a350cde6c678e9ed71beebe709a039a7ba233af6fcb76b5ef282a\": rpc error: code = NotFound desc = could not find container \"16b86f83287a350cde6c678e9ed71beebe709a039a7ba233af6fcb76b5ef282a\": container with ID starting with 16b86f83287a350cde6c678e9ed71beebe709a039a7ba233af6fcb76b5ef282a not found: ID does not exist"
Oct 30 22:14:53.086871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:53.086599    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"]
Oct 30 22:14:53.091440 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:53.091407    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-blssm"]
Oct 30 22:14:53.201046 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:53.201009    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"]
Oct 30 22:14:53.205540 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:53.205505    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-xftsg"]
Oct 30 22:14:53.336297 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:53.336258    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"]
Oct 30 22:14:53.340389 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:53.340309    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7fabe91d-openshift-default-bc7cfb59c-4rfqv"]
Oct 30 22:14:53.394369 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:53.394329    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f396aaf-e41b-49c4-8734-cd729fa95810" path="/var/lib/kubelet/pods/6f396aaf-e41b-49c4-8734-cd729fa95810/volumes"
Oct 30 22:14:53.394895 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:53.394874    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a268712-b551-4bdd-8fbd-70e124508a44" path="/var/lib/kubelet/pods/7a268712-b551-4bdd-8fbd-70e124508a44/volumes"
Oct 30 22:14:53.395414 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:53.395394    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a765ef62-b023-42d6-96c1-8e7715f11f3c" path="/var/lib/kubelet/pods/a765ef62-b023-42d6-96c1-8e7715f11f3c/volumes"
Oct 30 22:14:53.758785 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:53.758712    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:14:53.760208 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:53.760184    4553 patch_prober.go:29] interesting pod/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:14:53.760343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:53.760232    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn" podUID="38824b8a-8c76-412c-9d99-eb8955f8d237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 22:14:54.759328 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:54.759292    4553 patch_prober.go:29] interesting pod/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:14:54.759754 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:54.759347    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn" podUID="38824b8a-8c76-412c-9d99-eb8955f8d237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 22:14:55.758996 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:55.758967    4553 patch_prober.go:29] interesting pod/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:14:55.759217 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:55.759025    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn" podUID="38824b8a-8c76-412c-9d99-eb8955f8d237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 22:14:56.759601 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:56.759566    4553 patch_prober.go:29] interesting pod/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:14:56.760030 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:56.759626    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn" podUID="38824b8a-8c76-412c-9d99-eb8955f8d237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 22:14:57.759652 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:57.759616    4553 patch_prober.go:29] interesting pod/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:14:57.760097 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:57.759679    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn" podUID="38824b8a-8c76-412c-9d99-eb8955f8d237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Oct 30 22:14:58.204309 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:14:58.204273    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:14:58.766059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:58.766027    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:14:58.768172 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:58.768109    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:14:58.768919 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:14:58.768895    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:15:00.254768 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.254735    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"]
Oct 30 22:15:00.255623 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255599    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7a268712-b551-4bdd-8fbd-70e124508a44" containerName="istio-proxy"
Oct 30 22:15:00.255748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255624    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a268712-b551-4bdd-8fbd-70e124508a44" containerName="istio-proxy"
Oct 30 22:15:00.255748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255647    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a765ef62-b023-42d6-96c1-8e7715f11f3c" containerName="istio-proxy"
Oct 30 22:15:00.255748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255655    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a765ef62-b023-42d6-96c1-8e7715f11f3c" containerName="istio-proxy"
Oct 30 22:15:00.255748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255672    4553 cpu_manager.go:414] "containerMap: removing container" podUID="54fc71a0-ea97-4740-8f06-d41584d804dc" containerName="payload-pre-processing"
Oct 30 22:15:00.255748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255679    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54fc71a0-ea97-4740-8f06-d41584d804dc" containerName="payload-pre-processing"
Oct 30 22:15:00.255748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255706    4553 cpu_manager.go:414] "containerMap: removing container" podUID="6f396aaf-e41b-49c4-8734-cd729fa95810" containerName="istio-proxy"
Oct 30 22:15:00.255748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255713    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f396aaf-e41b-49c4-8734-cd729fa95810" containerName="istio-proxy"
Oct 30 22:15:00.255748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255732    4553 cpu_manager.go:414] "containerMap: removing container" podUID="f72b872a-65b7-4923-b5f9-fbb01be82e41" containerName="maas-api"
Oct 30 22:15:00.255748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255739    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f72b872a-65b7-4923-b5f9-fbb01be82e41" containerName="maas-api"
Oct 30 22:15:00.256241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255757    4553 cpu_manager.go:414] "containerMap: removing container" podUID="2d913198-413c-496e-b54f-a4823f53df45" containerName="payload-processing"
Oct 30 22:15:00.256241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255765    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d913198-413c-496e-b54f-a4823f53df45" containerName="payload-processing"
Oct 30 22:15:00.256241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255932    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a268712-b551-4bdd-8fbd-70e124508a44" containerName="istio-proxy"
Oct 30 22:15:00.256241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255946    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="a765ef62-b023-42d6-96c1-8e7715f11f3c" containerName="istio-proxy"
Oct 30 22:15:00.256241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255959    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="54fc71a0-ea97-4740-8f06-d41584d804dc" containerName="payload-pre-processing"
Oct 30 22:15:00.256241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255973    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f396aaf-e41b-49c4-8734-cd729fa95810" containerName="istio-proxy"
Oct 30 22:15:00.256241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255984    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="f72b872a-65b7-4923-b5f9-fbb01be82e41" containerName="maas-api"
Oct 30 22:15:00.256241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.255996    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d913198-413c-496e-b54f-a4823f53df45" containerName="payload-processing"
Oct 30 22:15:00.261328 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.261307    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"]
Oct 30 22:15:00.261463 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.261441    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"
Oct 30 22:15:00.265673 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.265653    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"
Oct 30 22:15:00.347574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.345828    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"]
Oct 30 22:15:00.350041 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.350008    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"]
Oct 30 22:15:00.373149 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.373117    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:15:00.403127 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.403082    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zljj4\" (UniqueName: \"kubernetes.io/projected/af14fa73-afe3-4deb-a197-4b91f1296b3c-kube-api-access-zljj4\") pod \"maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz\" (UID: \"af14fa73-afe3-4deb-a197-4b91f1296b3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"
Oct 30 22:15:00.403348 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.403174    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnwgp\" (UniqueName: \"kubernetes.io/projected/a42b60f3-ab04-493b-aeab-ca862f753d56-kube-api-access-jnwgp\") pod \"maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl\" (UID: \"a42b60f3-ab04-493b-aeab-ca862f753d56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"
Oct 30 22:15:00.403425 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.403368    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a42b60f3-ab04-493b-aeab-ca862f753d56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl\" (UID: \"a42b60f3-ab04-493b-aeab-ca862f753d56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"
Oct 30 22:15:00.403574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.403548    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af14fa73-afe3-4deb-a197-4b91f1296b3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz\" (UID: \"af14fa73-afe3-4deb-a197-4b91f1296b3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"
Oct 30 22:15:00.518107 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.517999    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljj4\" (UniqueName: \"kubernetes.io/projected/af14fa73-afe3-4deb-a197-4b91f1296b3c-kube-api-access-zljj4\") pod \"maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz\" (UID: \"af14fa73-afe3-4deb-a197-4b91f1296b3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"
Oct 30 22:15:00.523371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.523294    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnwgp\" (UniqueName: \"kubernetes.io/projected/a42b60f3-ab04-493b-aeab-ca862f753d56-kube-api-access-jnwgp\") pod \"maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl\" (UID: \"a42b60f3-ab04-493b-aeab-ca862f753d56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"
Oct 30 22:15:00.523601 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.523573    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a42b60f3-ab04-493b-aeab-ca862f753d56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl\" (UID: \"a42b60f3-ab04-493b-aeab-ca862f753d56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"
Oct 30 22:15:00.523736 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.523675    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af14fa73-afe3-4deb-a197-4b91f1296b3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz\" (UID: \"af14fa73-afe3-4deb-a197-4b91f1296b3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"
Oct 30 22:15:00.524595 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.524558    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af14fa73-afe3-4deb-a197-4b91f1296b3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz\" (UID: \"af14fa73-afe3-4deb-a197-4b91f1296b3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"
Oct 30 22:15:00.526698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.526676    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a42b60f3-ab04-493b-aeab-ca862f753d56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl\" (UID: \"a42b60f3-ab04-493b-aeab-ca862f753d56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"
Oct 30 22:15:00.583999 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.583970    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zljj4\" (UniqueName: \"kubernetes.io/projected/af14fa73-afe3-4deb-a197-4b91f1296b3c-kube-api-access-zljj4\") pod \"maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz\" (UID: \"af14fa73-afe3-4deb-a197-4b91f1296b3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"
Oct 30 22:15:00.584771 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.584754    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnwgp\" (UniqueName: \"kubernetes.io/projected/a42b60f3-ab04-493b-aeab-ca862f753d56-kube-api-access-jnwgp\") pod \"maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl\" (UID: \"a42b60f3-ab04-493b-aeab-ca862f753d56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"
Oct 30 22:15:00.626816 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.626781    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"
Oct 30 22:15:00.645199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:00.645175    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"
Oct 30 22:15:01.011547 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.011505    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"]
Oct 30 22:15:01.051919 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:15:01.051893    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42b60f3_ab04_493b_aeab_ca862f753d56.slice/crio-aea035f1f0756e539faf5a11ced825db0b9e542c8285fe59a4b80339df05deef WatchSource:0}: Error finding container aea035f1f0756e539faf5a11ced825db0b9e542c8285fe59a4b80339df05deef: Status 404 returned error can't find the container with id aea035f1f0756e539faf5a11ced825db0b9e542c8285fe59a4b80339df05deef
Oct 30 22:15:01.064471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.064443    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"]
Oct 30 22:15:01.065196 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:15:01.065171    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf14fa73_afe3_4deb_a197_4b91f1296b3c.slice/crio-8bad8d41d2b1183fa096edf7d6e75bb9d401c00a01b2912dd41792ef1495ae5c WatchSource:0}: Error finding container 8bad8d41d2b1183fa096edf7d6e75bb9d401c00a01b2912dd41792ef1495ae5c: Status 404 returned error can't find the container with id 8bad8d41d2b1183fa096edf7d6e75bb9d401c00a01b2912dd41792ef1495ae5c
Oct 30 22:15:01.171970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.171939    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"]
Oct 30 22:15:01.177385 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.177362    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"
Oct 30 22:15:01.231745 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.231715    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-859078\""
Oct 30 22:15:01.232473 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.232456    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-859078-dockercfg-m5hm7\""
Oct 30 22:15:01.241072 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.241043    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"]
Oct 30 22:15:01.259626 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.259591    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39819f5-d459-4491-a6cd-eab808ff9b36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9\" (UID: \"b39819f5-d459-4491-a6cd-eab808ff9b36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"
Oct 30 22:15:01.260162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.259797    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2gnm\" (UniqueName: \"kubernetes.io/projected/b39819f5-d459-4491-a6cd-eab808ff9b36-kube-api-access-j2gnm\") pod \"payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9\" (UID: \"b39819f5-d459-4491-a6cd-eab808ff9b36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"
Oct 30 22:15:01.360677 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.360642    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2gnm\" (UniqueName: \"kubernetes.io/projected/b39819f5-d459-4491-a6cd-eab808ff9b36-kube-api-access-j2gnm\") pod \"payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9\" (UID: \"b39819f5-d459-4491-a6cd-eab808ff9b36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"
Oct 30 22:15:01.360890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.360770    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39819f5-d459-4491-a6cd-eab808ff9b36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9\" (UID: \"b39819f5-d459-4491-a6cd-eab808ff9b36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"
Oct 30 22:15:01.361571 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.361543    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39819f5-d459-4491-a6cd-eab808ff9b36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9\" (UID: \"b39819f5-d459-4491-a6cd-eab808ff9b36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"
Oct 30 22:15:01.419640 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.419607    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2gnm\" (UniqueName: \"kubernetes.io/projected/b39819f5-d459-4491-a6cd-eab808ff9b36-kube-api-access-j2gnm\") pod \"payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9\" (UID: \"b39819f5-d459-4491-a6cd-eab808ff9b36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"
Oct 30 22:15:01.492845 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.492808    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"
Oct 30 22:15:01.871349 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.871318    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"]
Oct 30 22:15:01.872885 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:15:01.872856    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39819f5_d459_4491_a6cd_eab808ff9b36.slice/crio-85b678f82abc1280c70e8c231e76c92a4f726152d4a266385ba6e8f260e9411c WatchSource:0}: Error finding container 85b678f82abc1280c70e8c231e76c92a4f726152d4a266385ba6e8f260e9411c: Status 404 returned error can't find the container with id 85b678f82abc1280c70e8c231e76c92a4f726152d4a266385ba6e8f260e9411c
Oct 30 22:15:01.930682 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.930655    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"]
Oct 30 22:15:01.935739 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.935724    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"
Oct 30 22:15:01.968393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.968352    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b277a651-334b-4560-9dc9-89bf75a1b9a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-859078-795cf98998-mghsr\" (UID: \"b277a651-334b-4560-9dc9-89bf75a1b9a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"
Oct 30 22:15:01.968575 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.968541    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw4b6\" (UniqueName: \"kubernetes.io/projected/b277a651-334b-4560-9dc9-89bf75a1b9a6-kube-api-access-bw4b6\") pod \"payload-processing-e2e-ipp-b-859078-795cf98998-mghsr\" (UID: \"b277a651-334b-4560-9dc9-89bf75a1b9a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"
Oct 30 22:15:01.991981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.991944    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz" event={"ID":"af14fa73-afe3-4deb-a197-4b91f1296b3c","Type":"ContainerStarted","Data":"8bad8d41d2b1183fa096edf7d6e75bb9d401c00a01b2912dd41792ef1495ae5c"}
Oct 30 22:15:01.994200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.994128    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9" event={"ID":"b39819f5-d459-4491-a6cd-eab808ff9b36","Type":"ContainerStarted","Data":"6e33f8ad3c575108a4c8c619380eb4eeafb47fdc243e823ff66b30eaffa5c551"}
Oct 30 22:15:01.994362 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.994226    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9" event={"ID":"b39819f5-d459-4491-a6cd-eab808ff9b36","Type":"ContainerStarted","Data":"85b678f82abc1280c70e8c231e76c92a4f726152d4a266385ba6e8f260e9411c"}
Oct 30 22:15:01.994362 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.994253    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"
Oct 30 22:15:01.994477 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.994457    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"]
Oct 30 22:15:01.995662 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:01.995639    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl" event={"ID":"a42b60f3-ab04-493b-aeab-ca862f753d56","Type":"ContainerStarted","Data":"aea035f1f0756e539faf5a11ced825db0b9e542c8285fe59a4b80339df05deef"}
Oct 30 22:15:02.070023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:02.069986    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b277a651-334b-4560-9dc9-89bf75a1b9a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-859078-795cf98998-mghsr\" (UID: \"b277a651-334b-4560-9dc9-89bf75a1b9a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"
Oct 30 22:15:02.070258 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:02.070125    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw4b6\" (UniqueName: \"kubernetes.io/projected/b277a651-334b-4560-9dc9-89bf75a1b9a6-kube-api-access-bw4b6\") pod \"payload-processing-e2e-ipp-b-859078-795cf98998-mghsr\" (UID: \"b277a651-334b-4560-9dc9-89bf75a1b9a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"
Oct 30 22:15:02.070826 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:02.070796    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b277a651-334b-4560-9dc9-89bf75a1b9a6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-859078-795cf98998-mghsr\" (UID: \"b277a651-334b-4560-9dc9-89bf75a1b9a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"
Oct 30 22:15:02.129477 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:02.129398    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw4b6\" (UniqueName: \"kubernetes.io/projected/b277a651-334b-4560-9dc9-89bf75a1b9a6-kube-api-access-bw4b6\") pod \"payload-processing-e2e-ipp-b-859078-795cf98998-mghsr\" (UID: \"b277a651-334b-4560-9dc9-89bf75a1b9a6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"
Oct 30 22:15:02.175747 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:02.175684    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9" podStartSLOduration=1.175665822 podStartE2EDuration="1.175665822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.173433936 +0000 UTC m=+1952.907796968" watchObservedRunningTime="2026-10-30 22:15:02.175665822 +0000 UTC m=+1952.910028854"
Oct 30 22:15:02.248972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:02.248941    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"
Oct 30 22:15:02.667954 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:02.656033    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"]
Oct 30 22:15:03.008424 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:03.007771    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr" event={"ID":"b277a651-334b-4560-9dc9-89bf75a1b9a6","Type":"ContainerStarted","Data":"f2ed814b5f12dec9bdbe496de1d3148f0df05a1a8438acb91a8cc313df388bad"}
Oct 30 22:15:03.008424 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:03.007834    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr" event={"ID":"b277a651-334b-4560-9dc9-89bf75a1b9a6","Type":"ContainerStarted","Data":"dc688b753ebe9b51068322d65bcb910b95a8ecaeeb29f0ad97b380f09ce43779"}
Oct 30 22:15:03.008424 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:03.007886    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"
Oct 30 22:15:03.126332 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:03.126275    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr" podStartSLOduration=2.126256132 podStartE2EDuration="2.126256132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.123856391 +0000 UTC m=+1953.858219424" watchObservedRunningTime="2026-10-30 22:15:03.126256132 +0000 UTC m=+1953.860619162"
Oct 30 22:15:06.035284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:06.035250    4553 generic.go:356] "Generic (PLEG): container finished" podID="af14fa73-afe3-4deb-a197-4b91f1296b3c" containerID="b1cbff4deaff247dc6477b99533ac50bad4ccda62a28dca5a9e5225317b37bee" exitCode=60
Oct 30 22:15:06.035667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:06.035329    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz" event={"ID":"af14fa73-afe3-4deb-a197-4b91f1296b3c","Type":"ContainerDied","Data":"b1cbff4deaff247dc6477b99533ac50bad4ccda62a28dca5a9e5225317b37bee"}
Oct 30 22:15:06.035667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:06.035620    4553 scope.go:122] "RemoveContainer" containerID="b1cbff4deaff247dc6477b99533ac50bad4ccda62a28dca5a9e5225317b37bee"
Oct 30 22:15:06.037370 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:06.037333    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl" event={"ID":"a42b60f3-ab04-493b-aeab-ca862f753d56","Type":"ContainerStarted","Data":"5dca7346ad80f880ecdc7d1ab87e999914b7601014deb912e7a8c0fbe2dc77c8"}
Oct 30 22:15:06.270440 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:06.270387    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl" podStartSLOduration=1.967855363 podStartE2EDuration="6.270369137s" totalImagesPullingTime="4.302513774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:06.26734847 +0000 UTC m=+1957.001711511" watchObservedRunningTime="2026-10-30 22:15:06.270369137 +0000 UTC m=+1957.004732168"
Oct 30 22:15:07.046934 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:07.046895    4553 generic.go:356] "Generic (PLEG): container finished" podID="af14fa73-afe3-4deb-a197-4b91f1296b3c" containerID="de6f74b1649b6aeabf40c989ede0e220ebfb630f4d1f413b332ba06f961ae93a" exitCode=60
Oct 30 22:15:07.047474 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:07.046986    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz" event={"ID":"af14fa73-afe3-4deb-a197-4b91f1296b3c","Type":"ContainerDied","Data":"de6f74b1649b6aeabf40c989ede0e220ebfb630f4d1f413b332ba06f961ae93a"}
Oct 30 22:15:07.047474 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:07.047118    4553 scope.go:122] "RemoveContainer" containerID="b1cbff4deaff247dc6477b99533ac50bad4ccda62a28dca5a9e5225317b37bee"
Oct 30 22:15:07.047584 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:07.047510    4553 scope.go:122] "RemoveContainer" containerID="de6f74b1649b6aeabf40c989ede0e220ebfb630f4d1f413b332ba06f961ae93a"
Oct 30 22:15:07.047912 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:15:07.047885    4553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz_odh-ai-gateway-infra(af14fa73-afe3-4deb-a197-4b91f1296b3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz" podUID="af14fa73-afe3-4deb-a197-4b91f1296b3c"
Oct 30 22:15:07.048996 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:07.048972    4553 generic.go:356] "Generic (PLEG): container finished" podID="a42b60f3-ab04-493b-aeab-ca862f753d56" containerID="5dca7346ad80f880ecdc7d1ab87e999914b7601014deb912e7a8c0fbe2dc77c8" exitCode=60
Oct 30 22:15:07.049093 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:07.049004    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl" event={"ID":"a42b60f3-ab04-493b-aeab-ca862f753d56","Type":"ContainerDied","Data":"5dca7346ad80f880ecdc7d1ab87e999914b7601014deb912e7a8c0fbe2dc77c8"}
Oct 30 22:15:07.049389 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:07.049374    4553 scope.go:122] "RemoveContainer" containerID="5dca7346ad80f880ecdc7d1ab87e999914b7601014deb912e7a8c0fbe2dc77c8"
Oct 30 22:15:08.058777 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:08.058752    4553 scope.go:122] "RemoveContainer" containerID="de6f74b1649b6aeabf40c989ede0e220ebfb630f4d1f413b332ba06f961ae93a"
Oct 30 22:15:08.059311 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:15:08.059129    4553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz_odh-ai-gateway-infra(af14fa73-afe3-4deb-a197-4b91f1296b3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz" podUID="af14fa73-afe3-4deb-a197-4b91f1296b3c"
Oct 30 22:15:08.060623 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:08.060599    4553 generic.go:356] "Generic (PLEG): container finished" podID="a42b60f3-ab04-493b-aeab-ca862f753d56" containerID="f96f782731d2ea4c54124d932fb68addb874e55003b9d5bb76a329ee26f01409" exitCode=60
Oct 30 22:15:08.060779 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:08.060656    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl" event={"ID":"a42b60f3-ab04-493b-aeab-ca862f753d56","Type":"ContainerDied","Data":"f96f782731d2ea4c54124d932fb68addb874e55003b9d5bb76a329ee26f01409"}
Oct 30 22:15:08.060779 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:08.060704    4553 scope.go:122] "RemoveContainer" containerID="5dca7346ad80f880ecdc7d1ab87e999914b7601014deb912e7a8c0fbe2dc77c8"
Oct 30 22:15:08.060995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:08.060976    4553 scope.go:122] "RemoveContainer" containerID="f96f782731d2ea4c54124d932fb68addb874e55003b9d5bb76a329ee26f01409"
Oct 30 22:15:08.061291 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:15:08.061270    4553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl_odh-ai-gateway-infra(a42b60f3-ab04-493b-aeab-ca862f753d56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl" podUID="a42b60f3-ab04-493b-aeab-ca862f753d56"
Oct 30 22:15:08.442376 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:08.442335    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"]
Oct 30 22:15:08.442626 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:08.442602    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq" podUID="cae0b720-525f-43fc-a94b-42e079ce85b4" containerName="istio-proxy" containerID="cri-o://3db1554d97fd3c808a50fc26d55128136b9c835a49789fad86f73b18d046d0ca" gracePeriod=30
Oct 30 22:15:09.069442 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:09.069217    4553 scope.go:122] "RemoveContainer" containerID="f96f782731d2ea4c54124d932fb68addb874e55003b9d5bb76a329ee26f01409"
Oct 30 22:15:09.070371 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:15:09.070340    4553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl_odh-ai-gateway-infra(a42b60f3-ab04-493b-aeab-ca862f753d56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl" podUID="a42b60f3-ab04-493b-aeab-ca862f753d56"
Oct 30 22:15:13.009606 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.009576    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"
Oct 30 22:15:13.718418 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.718350    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:15:13.836761 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.836724    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-envoy\") pod \"cae0b720-525f-43fc-a94b-42e079ce85b4\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") "
Oct 30 22:15:13.836965 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.836813    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-data\") pod \"cae0b720-525f-43fc-a94b-42e079ce85b4\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") "
Oct 30 22:15:13.836965 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.836849    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-podinfo\") pod \"cae0b720-525f-43fc-a94b-42e079ce85b4\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") "
Oct 30 22:15:13.836965 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.836951    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae0b720-525f-43fc-a94b-42e079ce85b4-kube-api-access-r2qpt\" (UniqueName: \"kubernetes.io/projected/cae0b720-525f-43fc-a94b-42e079ce85b4-kube-api-access-r2qpt\") pod \"cae0b720-525f-43fc-a94b-42e079ce85b4\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") "
Oct 30 22:15:13.837119 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.837045    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-workload-socket\") pod \"cae0b720-525f-43fc-a94b-42e079ce85b4\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") "
Oct 30 22:15:13.837119 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.837098    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-token\" (UniqueName: \"kubernetes.io/projected/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-token\") pod \"cae0b720-525f-43fc-a94b-42e079ce85b4\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") "
Oct 30 22:15:13.837282 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.837120    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-data" pod "cae0b720-525f-43fc-a94b-42e079ce85b4" (UID: "cae0b720-525f-43fc-a94b-42e079ce85b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:13.837282 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.837170    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-workload-certs\") pod \"cae0b720-525f-43fc-a94b-42e079ce85b4\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") "
Oct 30 22:15:13.837282 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.837246    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-credential-socket\") pod \"cae0b720-525f-43fc-a94b-42e079ce85b4\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") "
Oct 30 22:15:13.837282 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.837277    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cae0b720-525f-43fc-a94b-42e079ce85b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae0b720-525f-43fc-a94b-42e079ce85b4-istiod-ca-cert\") pod \"cae0b720-525f-43fc-a94b-42e079ce85b4\" (UID: \"cae0b720-525f-43fc-a94b-42e079ce85b4\") "
Oct 30 22:15:13.837472 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.837358    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-workload-socket" pod "cae0b720-525f-43fc-a94b-42e079ce85b4" (UID: "cae0b720-525f-43fc-a94b-42e079ce85b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:13.837522 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.837479    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-workload-certs" pod "cae0b720-525f-43fc-a94b-42e079ce85b4" (UID: "cae0b720-525f-43fc-a94b-42e079ce85b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:13.837767 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.837728    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-credential-socket" pod "cae0b720-525f-43fc-a94b-42e079ce85b4" (UID: "cae0b720-525f-43fc-a94b-42e079ce85b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:13.837916 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.837797    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:13.837916 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.837812    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:13.837916 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.837826    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:13.837916 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.837840    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:13.838158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.837948    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae0b720-525f-43fc-a94b-42e079ce85b4-istiod-ca-cert" pod "cae0b720-525f-43fc-a94b-42e079ce85b4" (UID: "cae0b720-525f-43fc-a94b-42e079ce85b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:13.839231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.839209    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-envoy" pod "cae0b720-525f-43fc-a94b-42e079ce85b4" (UID: "cae0b720-525f-43fc-a94b-42e079ce85b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:13.839444 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.839420    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-podinfo" pod "cae0b720-525f-43fc-a94b-42e079ce85b4" (UID: "cae0b720-525f-43fc-a94b-42e079ce85b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:15:13.839782 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.839762    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae0b720-525f-43fc-a94b-42e079ce85b4-kube-api-access-r2qpt" pod "cae0b720-525f-43fc-a94b-42e079ce85b4" (UID: "cae0b720-525f-43fc-a94b-42e079ce85b4"). InnerVolumeSpecName "kube-api-access-r2qpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:13.839876 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.839796    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-token" pod "cae0b720-525f-43fc-a94b-42e079ce85b4" (UID: "cae0b720-525f-43fc-a94b-42e079ce85b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:13.938784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.938747    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:13.938784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.938778    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2qpt\" (UniqueName: \"kubernetes.io/projected/cae0b720-525f-43fc-a94b-42e079ce85b4-kube-api-access-r2qpt\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:13.938784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.938790    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:13.939022 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.938798    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cae0b720-525f-43fc-a94b-42e079ce85b4-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:13.939022 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:13.938807    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cae0b720-525f-43fc-a94b-42e079ce85b4-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:14.114826 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:14.114792    4553 generic.go:356] "Generic (PLEG): container finished" podID="cae0b720-525f-43fc-a94b-42e079ce85b4" containerID="3db1554d97fd3c808a50fc26d55128136b9c835a49789fad86f73b18d046d0ca" exitCode=0
Oct 30 22:15:14.115367 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:14.114839    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"
Oct 30 22:15:14.115367 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:14.114876    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq" event={"ID":"cae0b720-525f-43fc-a94b-42e079ce85b4","Type":"ContainerDied","Data":"3db1554d97fd3c808a50fc26d55128136b9c835a49789fad86f73b18d046d0ca"}
Oct 30 22:15:14.115367 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:14.114947    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq" event={"ID":"cae0b720-525f-43fc-a94b-42e079ce85b4","Type":"ContainerDied","Data":"9e4e54d969129e4c732d8f16e398824ebdcaba37770a6089d455a7400320fd98"}
Oct 30 22:15:14.115367 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:14.114982    4553 scope.go:122] "RemoveContainer" containerID="3db1554d97fd3c808a50fc26d55128136b9c835a49789fad86f73b18d046d0ca"
Oct 30 22:15:14.125882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:14.125861    4553 scope.go:122] "RemoveContainer" containerID="81d22899ad6575a630c03418d4ce4d82410837e40ed6c76cb1900ccfbd55e7ff"
Oct 30 22:15:14.135373 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:14.135354    4553 scope.go:122] "RemoveContainer" containerID="3db1554d97fd3c808a50fc26d55128136b9c835a49789fad86f73b18d046d0ca"
Oct 30 22:15:14.135701 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:15:14.135681    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db1554d97fd3c808a50fc26d55128136b9c835a49789fad86f73b18d046d0ca\": container with ID starting with 3db1554d97fd3c808a50fc26d55128136b9c835a49789fad86f73b18d046d0ca not found: ID does not exist" containerID="3db1554d97fd3c808a50fc26d55128136b9c835a49789fad86f73b18d046d0ca"
Oct 30 22:15:14.135789 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:14.135714    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db1554d97fd3c808a50fc26d55128136b9c835a49789fad86f73b18d046d0ca"} err="failed to get container status \"3db1554d97fd3c808a50fc26d55128136b9c835a49789fad86f73b18d046d0ca\": rpc error: code = NotFound desc = could not find container \"3db1554d97fd3c808a50fc26d55128136b9c835a49789fad86f73b18d046d0ca\": container with ID starting with 3db1554d97fd3c808a50fc26d55128136b9c835a49789fad86f73b18d046d0ca not found: ID does not exist"
Oct 30 22:15:14.135789 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:14.135740    4553 scope.go:122] "RemoveContainer" containerID="81d22899ad6575a630c03418d4ce4d82410837e40ed6c76cb1900ccfbd55e7ff"
Oct 30 22:15:14.136085 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:15:14.136065    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d22899ad6575a630c03418d4ce4d82410837e40ed6c76cb1900ccfbd55e7ff\": container with ID starting with 81d22899ad6575a630c03418d4ce4d82410837e40ed6c76cb1900ccfbd55e7ff not found: ID does not exist" containerID="81d22899ad6575a630c03418d4ce4d82410837e40ed6c76cb1900ccfbd55e7ff"
Oct 30 22:15:14.136128 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:14.136092    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d22899ad6575a630c03418d4ce4d82410837e40ed6c76cb1900ccfbd55e7ff"} err="failed to get container status \"81d22899ad6575a630c03418d4ce4d82410837e40ed6c76cb1900ccfbd55e7ff\": rpc error: code = NotFound desc = could not find container \"81d22899ad6575a630c03418d4ce4d82410837e40ed6c76cb1900ccfbd55e7ff\": container with ID starting with 81d22899ad6575a630c03418d4ce4d82410837e40ed6c76cb1900ccfbd55e7ff not found: ID does not exist"
Oct 30 22:15:14.285211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:14.285172    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"]
Oct 30 22:15:14.335899 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:14.335859    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2bc5e3-openshift-default-56d8586b4c-xw7zq"]
Oct 30 22:15:15.028644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:15.028614    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"
Oct 30 22:15:15.394517 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:15.394484    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cae0b720-525f-43fc-a94b-42e079ce85b4" path="/var/lib/kubelet/pods/cae0b720-525f-43fc-a94b-42e079ce85b4/volumes"
Oct 30 22:15:18.386099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:18.386060    4553 scope.go:122] "RemoveContainer" containerID="de6f74b1649b6aeabf40c989ede0e220ebfb630f4d1f413b332ba06f961ae93a"
Oct 30 22:15:19.033363 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.033278    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"]
Oct 30 22:15:19.033917 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.033902    4553 cpu_manager.go:414] "containerMap: removing container" podUID="cae0b720-525f-43fc-a94b-42e079ce85b4" containerName="istio-proxy"
Oct 30 22:15:19.033963 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.033919    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cae0b720-525f-43fc-a94b-42e079ce85b4" containerName="istio-proxy"
Oct 30 22:15:19.033963 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.033943    4553 cpu_manager.go:414] "containerMap: removing container" podUID="cae0b720-525f-43fc-a94b-42e079ce85b4" containerName="istio-proxy"
Oct 30 22:15:19.033963 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.033949    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cae0b720-525f-43fc-a94b-42e079ce85b4" containerName="istio-proxy"
Oct 30 22:15:19.034062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.034052    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="cae0b720-525f-43fc-a94b-42e079ce85b4" containerName="istio-proxy"
Oct 30 22:15:19.034099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.034067    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="cae0b720-525f-43fc-a94b-42e079ce85b4" containerName="istio-proxy"
Oct 30 22:15:19.037695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.037673    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"
Oct 30 22:15:19.093834 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.093799    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"]
Oct 30 22:15:19.099757 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.099722    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ca1e4a-16cc-44cf-b5a0-86e65294706e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn\" (UID: \"43ca1e4a-16cc-44cf-b5a0-86e65294706e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"
Oct 30 22:15:19.100016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.099995    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lpfw\" (UniqueName: \"kubernetes.io/projected/43ca1e4a-16cc-44cf-b5a0-86e65294706e-kube-api-access-2lpfw\") pod \"payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn\" (UID: \"43ca1e4a-16cc-44cf-b5a0-86e65294706e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"
Oct 30 22:15:19.140081 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.139841    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2edb49-dockercfg-6qdvk\""
Oct 30 22:15:19.140081 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.139919    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2edb49\""
Oct 30 22:15:19.229531 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.229466    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lpfw\" (UniqueName: \"kubernetes.io/projected/43ca1e4a-16cc-44cf-b5a0-86e65294706e-kube-api-access-2lpfw\") pod \"payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn\" (UID: \"43ca1e4a-16cc-44cf-b5a0-86e65294706e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"
Oct 30 22:15:19.229736 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.229587    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ca1e4a-16cc-44cf-b5a0-86e65294706e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn\" (UID: \"43ca1e4a-16cc-44cf-b5a0-86e65294706e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"
Oct 30 22:15:19.235163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.230586    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ca1e4a-16cc-44cf-b5a0-86e65294706e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn\" (UID: \"43ca1e4a-16cc-44cf-b5a0-86e65294706e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"
Oct 30 22:15:19.251176 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.251093    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"]
Oct 30 22:15:19.256088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.256057    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"
Oct 30 22:15:19.257400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.257367    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz" event={"ID":"af14fa73-afe3-4deb-a197-4b91f1296b3c","Type":"ContainerStarted","Data":"f572fd69e3c11143ff9b2a068adc21c050df8b4f3e4ed3f1d4a4238d760268b8"}
Oct 30 22:15:19.269898 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.269868    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"]
Oct 30 22:15:19.289071 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.288991    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lpfw\" (UniqueName: \"kubernetes.io/projected/43ca1e4a-16cc-44cf-b5a0-86e65294706e-kube-api-access-2lpfw\") pod \"payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn\" (UID: \"43ca1e4a-16cc-44cf-b5a0-86e65294706e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"
Oct 30 22:15:19.352996 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.352956    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"
Oct 30 22:15:19.435317 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.432742    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9214ee9-5a57-41c0-b744-7468092863cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns\" (UID: \"c9214ee9-5a57-41c0-b744-7468092863cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"
Oct 30 22:15:19.435317 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.432788    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tc9w\" (UniqueName: \"kubernetes.io/projected/c9214ee9-5a57-41c0-b744-7468092863cd-kube-api-access-8tc9w\") pod \"payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns\" (UID: \"c9214ee9-5a57-41c0-b744-7468092863cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"
Oct 30 22:15:19.490919 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.490860    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz" podStartSLOduration=15.299948820000001 podStartE2EDuration="19.490845105s" totalImagesPullingTime="4.190896285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:19.488966366 +0000 UTC m=+1970.223329399" watchObservedRunningTime="2026-10-30 22:15:19.490845105 +0000 UTC m=+1970.225208136"
Oct 30 22:15:19.534356 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.534312    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9214ee9-5a57-41c0-b744-7468092863cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns\" (UID: \"c9214ee9-5a57-41c0-b744-7468092863cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"
Oct 30 22:15:19.534569 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.534378    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tc9w\" (UniqueName: \"kubernetes.io/projected/c9214ee9-5a57-41c0-b744-7468092863cd-kube-api-access-8tc9w\") pod \"payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns\" (UID: \"c9214ee9-5a57-41c0-b744-7468092863cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"
Oct 30 22:15:19.535030 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.535010    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9214ee9-5a57-41c0-b744-7468092863cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns\" (UID: \"c9214ee9-5a57-41c0-b744-7468092863cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"
Oct 30 22:15:19.595714 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.595674    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tc9w\" (UniqueName: \"kubernetes.io/projected/c9214ee9-5a57-41c0-b744-7468092863cd-kube-api-access-8tc9w\") pod \"payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns\" (UID: \"c9214ee9-5a57-41c0-b744-7468092863cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"
Oct 30 22:15:19.731474 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.731441    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"]
Oct 30 22:15:19.732838 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:15:19.732813    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ca1e4a_16cc_44cf_b5a0_86e65294706e.slice/crio-906b35af86eee1d203d4fac8b34893c2b122eb8cc9c034d5847b18ca572c8be5 WatchSource:0}: Error finding container 906b35af86eee1d203d4fac8b34893c2b122eb8cc9c034d5847b18ca572c8be5: Status 404 returned error can't find the container with id 906b35af86eee1d203d4fac8b34893c2b122eb8cc9c034d5847b18ca572c8be5
Oct 30 22:15:19.879352 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:19.879255    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"
Oct 30 22:15:20.265778 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:20.265749    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"]
Oct 30 22:15:20.269527 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:20.269498    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn" event={"ID":"43ca1e4a-16cc-44cf-b5a0-86e65294706e","Type":"ContainerStarted","Data":"2d83a7d0874f4ac8bc7d06ec471bfb0376ec25fc360d56d9e91d6145bcb89054"}
Oct 30 22:15:20.269638 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:20.269550    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn" event={"ID":"43ca1e4a-16cc-44cf-b5a0-86e65294706e","Type":"ContainerStarted","Data":"906b35af86eee1d203d4fac8b34893c2b122eb8cc9c034d5847b18ca572c8be5"}
Oct 30 22:15:20.269638 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:20.269578    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"
Oct 30 22:15:20.271512 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:20.271492    4553 generic.go:356] "Generic (PLEG): container finished" podID="af14fa73-afe3-4deb-a197-4b91f1296b3c" containerID="f572fd69e3c11143ff9b2a068adc21c050df8b4f3e4ed3f1d4a4238d760268b8" exitCode=60
Oct 30 22:15:20.271620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:20.271564    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz" event={"ID":"af14fa73-afe3-4deb-a197-4b91f1296b3c","Type":"ContainerDied","Data":"f572fd69e3c11143ff9b2a068adc21c050df8b4f3e4ed3f1d4a4238d760268b8"}
Oct 30 22:15:20.271833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:20.271636    4553 scope.go:122] "RemoveContainer" containerID="de6f74b1649b6aeabf40c989ede0e220ebfb630f4d1f413b332ba06f961ae93a"
Oct 30 22:15:20.271958 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:20.271944    4553 scope.go:122] "RemoveContainer" containerID="f572fd69e3c11143ff9b2a068adc21c050df8b4f3e4ed3f1d4a4238d760268b8"
Oct 30 22:15:20.272240 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:15:20.272219    4553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz_odh-ai-gateway-infra(af14fa73-afe3-4deb-a197-4b91f1296b3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz" podUID="af14fa73-afe3-4deb-a197-4b91f1296b3c"
Oct 30 22:15:20.393230 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:20.393164    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn" podStartSLOduration=2.393123245 podStartE2EDuration="2.393123245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:20.388710547 +0000 UTC m=+1971.123073579" watchObservedRunningTime="2026-10-30 22:15:20.393123245 +0000 UTC m=+1971.127486277"
Oct 30 22:15:20.529911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:20.527377    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"]
Oct 30 22:15:21.282683 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:21.282650    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns" event={"ID":"c9214ee9-5a57-41c0-b744-7468092863cd","Type":"ContainerStarted","Data":"3d2c16d849d4300fecbe57f794eb0339fa587c381dfd5da9a8c015548a440b87"}
Oct 30 22:15:21.282836 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:21.282709    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns" event={"ID":"c9214ee9-5a57-41c0-b744-7468092863cd","Type":"ContainerStarted","Data":"61635dc0bed0cec0d4cfb1b1c4bdda448b27aa3167cee4a8a21ec8e20b46ebde"}
Oct 30 22:15:21.282836 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:21.282747    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"
Oct 30 22:15:21.386262 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:21.386231    4553 scope.go:122] "RemoveContainer" containerID="f96f782731d2ea4c54124d932fb68addb874e55003b9d5bb76a329ee26f01409"
Oct 30 22:15:21.450870 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:21.450847    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"
Oct 30 22:15:21.520557 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:21.520505    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns" podStartSLOduration=2.520490679 podStartE2EDuration="2.520490679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:21.51763576 +0000 UTC m=+1972.251998801" watchObservedRunningTime="2026-10-30 22:15:21.520490679 +0000 UTC m=+1972.254853710"
Oct 30 22:15:21.561937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:21.561858    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af14fa73-afe3-4deb-a197-4b91f1296b3c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af14fa73-afe3-4deb-a197-4b91f1296b3c-maas-api-service-ca\") pod \"af14fa73-afe3-4deb-a197-4b91f1296b3c\" (UID: \"af14fa73-afe3-4deb-a197-4b91f1296b3c\") "
Oct 30 22:15:21.561937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:21.561933    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af14fa73-afe3-4deb-a197-4b91f1296b3c-kube-api-access-zljj4\" (UniqueName: \"kubernetes.io/projected/af14fa73-afe3-4deb-a197-4b91f1296b3c-kube-api-access-zljj4\") pod \"af14fa73-afe3-4deb-a197-4b91f1296b3c\" (UID: \"af14fa73-afe3-4deb-a197-4b91f1296b3c\") "
Oct 30 22:15:21.562471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:21.562228    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af14fa73-afe3-4deb-a197-4b91f1296b3c-maas-api-service-ca" pod "af14fa73-afe3-4deb-a197-4b91f1296b3c" (UID: "af14fa73-afe3-4deb-a197-4b91f1296b3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:21.562471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:21.562382    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af14fa73-afe3-4deb-a197-4b91f1296b3c-maas-api-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.564281 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:21.564259    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af14fa73-afe3-4deb-a197-4b91f1296b3c-kube-api-access-zljj4" pod "af14fa73-afe3-4deb-a197-4b91f1296b3c" (UID: "af14fa73-afe3-4deb-a197-4b91f1296b3c"). InnerVolumeSpecName "kube-api-access-zljj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:21.664551 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:21.664518    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zljj4\" (UniqueName: \"kubernetes.io/projected/af14fa73-afe3-4deb-a197-4b91f1296b3c-kube-api-access-zljj4\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:22.290152 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:22.290101    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"
Oct 30 22:15:22.290362 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:22.290173    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz" event={"ID":"af14fa73-afe3-4deb-a197-4b91f1296b3c","Type":"ContainerDied","Data":"8bad8d41d2b1183fa096edf7d6e75bb9d401c00a01b2912dd41792ef1495ae5c"}
Oct 30 22:15:22.290362 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:22.290242    4553 scope.go:122] "RemoveContainer" containerID="f572fd69e3c11143ff9b2a068adc21c050df8b4f3e4ed3f1d4a4238d760268b8"
Oct 30 22:15:22.292533 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:22.292510    4553 generic.go:356] "Generic (PLEG): container finished" podID="a42b60f3-ab04-493b-aeab-ca862f753d56" containerID="1c45326755b3af8f9f2f6ca2face344453c9fb95a62e3fa0d6caca4a73d60f43" exitCode=60
Oct 30 22:15:22.292652 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:22.292578    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl" event={"ID":"a42b60f3-ab04-493b-aeab-ca862f753d56","Type":"ContainerDied","Data":"1c45326755b3af8f9f2f6ca2face344453c9fb95a62e3fa0d6caca4a73d60f43"}
Oct 30 22:15:22.293105 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:22.293090    4553 scope.go:122] "RemoveContainer" containerID="1c45326755b3af8f9f2f6ca2face344453c9fb95a62e3fa0d6caca4a73d60f43"
Oct 30 22:15:22.293387 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:15:22.293367    4553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl_odh-ai-gateway-infra(a42b60f3-ab04-493b-aeab-ca862f753d56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl" podUID="a42b60f3-ab04-493b-aeab-ca862f753d56"
Oct 30 22:15:22.300491 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:22.300471    4553 scope.go:122] "RemoveContainer" containerID="f96f782731d2ea4c54124d932fb68addb874e55003b9d5bb76a329ee26f01409"
Oct 30 22:15:22.458967 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:22.458931    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"]
Oct 30 22:15:22.466063 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:22.466038    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-69a1d2-29889975-jfwcz"]
Oct 30 22:15:23.399944 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:23.399908    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af14fa73-afe3-4deb-a197-4b91f1296b3c" path="/var/lib/kubelet/pods/af14fa73-afe3-4deb-a197-4b91f1296b3c/volumes"
Oct 30 22:15:23.617169 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:23.616721    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"]
Oct 30 22:15:23.804402 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:23.804365    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"
Oct 30 22:15:23.891934 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:23.891894    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a42b60f3-ab04-493b-aeab-ca862f753d56-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a42b60f3-ab04-493b-aeab-ca862f753d56-maas-api-service-ca\") pod \"a42b60f3-ab04-493b-aeab-ca862f753d56\" (UID: \"a42b60f3-ab04-493b-aeab-ca862f753d56\") "
Oct 30 22:15:23.892113 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:23.891947    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a42b60f3-ab04-493b-aeab-ca862f753d56-kube-api-access-jnwgp\" (UniqueName: \"kubernetes.io/projected/a42b60f3-ab04-493b-aeab-ca862f753d56-kube-api-access-jnwgp\") pod \"a42b60f3-ab04-493b-aeab-ca862f753d56\" (UID: \"a42b60f3-ab04-493b-aeab-ca862f753d56\") "
Oct 30 22:15:23.892291 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:23.892269    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42b60f3-ab04-493b-aeab-ca862f753d56-maas-api-service-ca" pod "a42b60f3-ab04-493b-aeab-ca862f753d56" (UID: "a42b60f3-ab04-493b-aeab-ca862f753d56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:23.892436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:23.892421    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a42b60f3-ab04-493b-aeab-ca862f753d56-maas-api-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:23.894330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:23.894310    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42b60f3-ab04-493b-aeab-ca862f753d56-kube-api-access-jnwgp" pod "a42b60f3-ab04-493b-aeab-ca862f753d56" (UID: "a42b60f3-ab04-493b-aeab-ca862f753d56"). InnerVolumeSpecName "kube-api-access-jnwgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:23.993783 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:23.993690    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnwgp\" (UniqueName: \"kubernetes.io/projected/a42b60f3-ab04-493b-aeab-ca862f753d56-kube-api-access-jnwgp\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:24.309594 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:24.309513    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"
Oct 30 22:15:24.309594 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:24.309541    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl" event={"ID":"a42b60f3-ab04-493b-aeab-ca862f753d56","Type":"ContainerDied","Data":"aea035f1f0756e539faf5a11ced825db0b9e542c8285fe59a4b80339df05deef"}
Oct 30 22:15:24.310351 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:24.309604    4553 scope.go:122] "RemoveContainer" containerID="1c45326755b3af8f9f2f6ca2face344453c9fb95a62e3fa0d6caca4a73d60f43"
Oct 30 22:15:24.477030 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:24.476996    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"]
Oct 30 22:15:24.527330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:24.527299    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7a54db-29889975-xbspl"]
Oct 30 22:15:25.393202 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:25.393160    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a42b60f3-ab04-493b-aeab-ca862f753d56" path="/var/lib/kubelet/pods/a42b60f3-ab04-493b-aeab-ca862f753d56/volumes"
Oct 30 22:15:27.298708 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:27.298666    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"]
Oct 30 22:15:27.410607 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:27.410572    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"]
Oct 30 22:15:28.202850 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:15:28.202813    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:15:31.286150 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:31.286110    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"
Oct 30 22:15:32.296293 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.296260    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"
Oct 30 22:15:32.299822 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.299785    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb" podUID="edb147b9-1a46-40c0-965b-69d0f9645055" containerName="payload-processing" containerID="cri-o://0ef843bcf814301e8af4f177f0467c71976888ec48dbf801a6d16a634133a546" gracePeriod=25
Oct 30 22:15:32.411299 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.411255    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl" podUID="a77804c9-9480-43ac-83ac-c9f81d27728e" containerName="payload-pre-processing" containerID="cri-o://66be530f5d89028ddb7787f10155fdf62fd858b8a0a6f9db7ce9fe7469bf2e20" gracePeriod=25
Oct 30 22:15:32.605597 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.605229    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"
Oct 30 22:15:32.689076 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.689046    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"
Oct 30 22:15:32.766509 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.766474    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb147b9-1a46-40c0-965b-69d0f9645055-kube-api-access-dhx4r\" (UniqueName: \"kubernetes.io/projected/edb147b9-1a46-40c0-965b-69d0f9645055-kube-api-access-dhx4r\") pod \"edb147b9-1a46-40c0-965b-69d0f9645055\" (UID: \"edb147b9-1a46-40c0-965b-69d0f9645055\") "
Oct 30 22:15:32.766707 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.766601    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edb147b9-1a46-40c0-965b-69d0f9645055-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edb147b9-1a46-40c0-965b-69d0f9645055-plugins-config-volume\") pod \"edb147b9-1a46-40c0-965b-69d0f9645055\" (UID: \"edb147b9-1a46-40c0-965b-69d0f9645055\") "
Oct 30 22:15:32.766985 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.766955    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb147b9-1a46-40c0-965b-69d0f9645055-plugins-config-volume" pod "edb147b9-1a46-40c0-965b-69d0f9645055" (UID: "edb147b9-1a46-40c0-965b-69d0f9645055"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:32.769017 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.768984    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb147b9-1a46-40c0-965b-69d0f9645055-kube-api-access-dhx4r" pod "edb147b9-1a46-40c0-965b-69d0f9645055" (UID: "edb147b9-1a46-40c0-965b-69d0f9645055"). InnerVolumeSpecName "kube-api-access-dhx4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:32.867759 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.867718    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a77804c9-9480-43ac-83ac-c9f81d27728e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77804c9-9480-43ac-83ac-c9f81d27728e-plugins-config-volume\") pod \"a77804c9-9480-43ac-83ac-c9f81d27728e\" (UID: \"a77804c9-9480-43ac-83ac-c9f81d27728e\") "
Oct 30 22:15:32.868082 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.868061    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a77804c9-9480-43ac-83ac-c9f81d27728e-kube-api-access-pqt6t\" (UniqueName: \"kubernetes.io/projected/a77804c9-9480-43ac-83ac-c9f81d27728e-kube-api-access-pqt6t\") pod \"a77804c9-9480-43ac-83ac-c9f81d27728e\" (UID: \"a77804c9-9480-43ac-83ac-c9f81d27728e\") "
Oct 30 22:15:32.868219 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.868099    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a77804c9-9480-43ac-83ac-c9f81d27728e-plugins-config-volume" pod "a77804c9-9480-43ac-83ac-c9f81d27728e" (UID: "a77804c9-9480-43ac-83ac-c9f81d27728e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:32.868579 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.868555    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhx4r\" (UniqueName: \"kubernetes.io/projected/edb147b9-1a46-40c0-965b-69d0f9645055-kube-api-access-dhx4r\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:32.868670 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.868585    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edb147b9-1a46-40c0-965b-69d0f9645055-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:32.868670 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.868600    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77804c9-9480-43ac-83ac-c9f81d27728e-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:32.870537 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.870506    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77804c9-9480-43ac-83ac-c9f81d27728e-kube-api-access-pqt6t" pod "a77804c9-9480-43ac-83ac-c9f81d27728e" (UID: "a77804c9-9480-43ac-83ac-c9f81d27728e"). InnerVolumeSpecName "kube-api-access-pqt6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:32.970303 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:32.970271    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqt6t\" (UniqueName: \"kubernetes.io/projected/a77804c9-9480-43ac-83ac-c9f81d27728e-kube-api-access-pqt6t\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:33.370620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.370584    4553 generic.go:356] "Generic (PLEG): container finished" podID="edb147b9-1a46-40c0-965b-69d0f9645055" containerID="0ef843bcf814301e8af4f177f0467c71976888ec48dbf801a6d16a634133a546" exitCode=0
Oct 30 22:15:33.371080 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.370631    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"
Oct 30 22:15:33.371080 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.370662    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb" event={"ID":"edb147b9-1a46-40c0-965b-69d0f9645055","Type":"ContainerDied","Data":"0ef843bcf814301e8af4f177f0467c71976888ec48dbf801a6d16a634133a546"}
Oct 30 22:15:33.371080 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.370721    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb" event={"ID":"edb147b9-1a46-40c0-965b-69d0f9645055","Type":"ContainerDied","Data":"90d3e9f52321ef4bf15575762f72cf8ab3d09e1b8a5428a143097c316071bc59"}
Oct 30 22:15:33.371080 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.370745    4553 scope.go:122] "RemoveContainer" containerID="0ef843bcf814301e8af4f177f0467c71976888ec48dbf801a6d16a634133a546"
Oct 30 22:15:33.372641 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.372616    4553 generic.go:356] "Generic (PLEG): container finished" podID="a77804c9-9480-43ac-83ac-c9f81d27728e" containerID="66be530f5d89028ddb7787f10155fdf62fd858b8a0a6f9db7ce9fe7469bf2e20" exitCode=0
Oct 30 22:15:33.372776 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.372695    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl" event={"ID":"a77804c9-9480-43ac-83ac-c9f81d27728e","Type":"ContainerDied","Data":"66be530f5d89028ddb7787f10155fdf62fd858b8a0a6f9db7ce9fe7469bf2e20"}
Oct 30 22:15:33.372776 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.372738    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl" event={"ID":"a77804c9-9480-43ac-83ac-c9f81d27728e","Type":"ContainerDied","Data":"ce157ae0cef5c1210673bd936e70e001b7637a6545cbda9d5d8c4d5eff725815"}
Oct 30 22:15:33.372900 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.372804    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"
Oct 30 22:15:33.383640 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.383614    4553 scope.go:122] "RemoveContainer" containerID="0ef843bcf814301e8af4f177f0467c71976888ec48dbf801a6d16a634133a546"
Oct 30 22:15:33.383956 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:15:33.383935    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef843bcf814301e8af4f177f0467c71976888ec48dbf801a6d16a634133a546\": container with ID starting with 0ef843bcf814301e8af4f177f0467c71976888ec48dbf801a6d16a634133a546 not found: ID does not exist" containerID="0ef843bcf814301e8af4f177f0467c71976888ec48dbf801a6d16a634133a546"
Oct 30 22:15:33.384033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.383966    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef843bcf814301e8af4f177f0467c71976888ec48dbf801a6d16a634133a546"} err="failed to get container status \"0ef843bcf814301e8af4f177f0467c71976888ec48dbf801a6d16a634133a546\": rpc error: code = NotFound desc = could not find container \"0ef843bcf814301e8af4f177f0467c71976888ec48dbf801a6d16a634133a546\": container with ID starting with 0ef843bcf814301e8af4f177f0467c71976888ec48dbf801a6d16a634133a546 not found: ID does not exist"
Oct 30 22:15:33.384033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.383984    4553 scope.go:122] "RemoveContainer" containerID="66be530f5d89028ddb7787f10155fdf62fd858b8a0a6f9db7ce9fe7469bf2e20"
Oct 30 22:15:33.393561 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.393540    4553 scope.go:122] "RemoveContainer" containerID="66be530f5d89028ddb7787f10155fdf62fd858b8a0a6f9db7ce9fe7469bf2e20"
Oct 30 22:15:33.393830 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:15:33.393812    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66be530f5d89028ddb7787f10155fdf62fd858b8a0a6f9db7ce9fe7469bf2e20\": container with ID starting with 66be530f5d89028ddb7787f10155fdf62fd858b8a0a6f9db7ce9fe7469bf2e20 not found: ID does not exist" containerID="66be530f5d89028ddb7787f10155fdf62fd858b8a0a6f9db7ce9fe7469bf2e20"
Oct 30 22:15:33.393891 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.393838    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66be530f5d89028ddb7787f10155fdf62fd858b8a0a6f9db7ce9fe7469bf2e20"} err="failed to get container status \"66be530f5d89028ddb7787f10155fdf62fd858b8a0a6f9db7ce9fe7469bf2e20\": rpc error: code = NotFound desc = could not find container \"66be530f5d89028ddb7787f10155fdf62fd858b8a0a6f9db7ce9fe7469bf2e20\": container with ID starting with 66be530f5d89028ddb7787f10155fdf62fd858b8a0a6f9db7ce9fe7469bf2e20 not found: ID does not exist"
Oct 30 22:15:33.537865 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.537829    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"]
Oct 30 22:15:33.544890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.544861    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-55456b66d7-6jzfl"]
Oct 30 22:15:33.653572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.653527    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"]
Oct 30 22:15:33.656356 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:33.656330    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a54db-58688547dc-jcngb"]
Oct 30 22:15:35.394949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:35.394912    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a77804c9-9480-43ac-83ac-c9f81d27728e" path="/var/lib/kubelet/pods/a77804c9-9480-43ac-83ac-c9f81d27728e/volumes"
Oct 30 22:15:35.395472 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:35.395456    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edb147b9-1a46-40c0-965b-69d0f9645055" path="/var/lib/kubelet/pods/edb147b9-1a46-40c0-965b-69d0f9645055/volumes"
Oct 30 22:15:36.672159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.671502    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"]
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672424    4553 cpu_manager.go:414] "containerMap: removing container" podUID="edb147b9-1a46-40c0-965b-69d0f9645055" containerName="payload-processing"
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672443    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb147b9-1a46-40c0-965b-69d0f9645055" containerName="payload-processing"
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672464    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a77804c9-9480-43ac-83ac-c9f81d27728e" containerName="payload-pre-processing"
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672472    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a77804c9-9480-43ac-83ac-c9f81d27728e" containerName="payload-pre-processing"
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672486    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a42b60f3-ab04-493b-aeab-ca862f753d56" containerName="cleanup"
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672493    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a42b60f3-ab04-493b-aeab-ca862f753d56" containerName="cleanup"
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672510    4553 cpu_manager.go:414] "containerMap: removing container" podUID="af14fa73-afe3-4deb-a197-4b91f1296b3c" containerName="cleanup"
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672517    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af14fa73-afe3-4deb-a197-4b91f1296b3c" containerName="cleanup"
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672535    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a42b60f3-ab04-493b-aeab-ca862f753d56" containerName="cleanup"
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672542    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a42b60f3-ab04-493b-aeab-ca862f753d56" containerName="cleanup"
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672553    4553 cpu_manager.go:414] "containerMap: removing container" podUID="af14fa73-afe3-4deb-a197-4b91f1296b3c" containerName="cleanup"
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672560    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af14fa73-afe3-4deb-a197-4b91f1296b3c" containerName="cleanup"
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672574    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a42b60f3-ab04-493b-aeab-ca862f753d56" containerName="cleanup"
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672580    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a42b60f3-ab04-493b-aeab-ca862f753d56" containerName="cleanup"
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672600    4553 cpu_manager.go:414] "containerMap: removing container" podUID="af14fa73-afe3-4deb-a197-4b91f1296b3c" containerName="cleanup"
Oct 30 22:15:36.672687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672607    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af14fa73-afe3-4deb-a197-4b91f1296b3c" containerName="cleanup"
Oct 30 22:15:36.673918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672738    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="af14fa73-afe3-4deb-a197-4b91f1296b3c" containerName="cleanup"
Oct 30 22:15:36.673918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672752    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="a42b60f3-ab04-493b-aeab-ca862f753d56" containerName="cleanup"
Oct 30 22:15:36.673918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672761    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="af14fa73-afe3-4deb-a197-4b91f1296b3c" containerName="cleanup"
Oct 30 22:15:36.673918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672771    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="a42b60f3-ab04-493b-aeab-ca862f753d56" containerName="cleanup"
Oct 30 22:15:36.673918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672782    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="af14fa73-afe3-4deb-a197-4b91f1296b3c" containerName="cleanup"
Oct 30 22:15:36.673918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672795    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb147b9-1a46-40c0-965b-69d0f9645055" containerName="payload-processing"
Oct 30 22:15:36.673918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.672805    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="a77804c9-9480-43ac-83ac-c9f81d27728e" containerName="payload-pre-processing"
Oct 30 22:15:36.679946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.679920    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"
Oct 30 22:15:36.720184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.720102    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"]
Oct 30 22:15:36.720563 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.720541    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbk5g\" (UniqueName: \"kubernetes.io/projected/3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce-kube-api-access-qbk5g\") pod \"payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682\" (UID: \"3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"
Oct 30 22:15:36.720684 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.720602    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682\" (UID: \"3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"
Oct 30 22:15:36.764094 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.764062    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-577ea8\""
Oct 30 22:15:36.764094 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.764062    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-577ea8-dockercfg-nnp98\""
Oct 30 22:15:36.822344 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.822303    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbk5g\" (UniqueName: \"kubernetes.io/projected/3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce-kube-api-access-qbk5g\") pod \"payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682\" (UID: \"3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"
Oct 30 22:15:36.822559 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.822356    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682\" (UID: \"3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"
Oct 30 22:15:36.823075 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.823053    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682\" (UID: \"3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"
Oct 30 22:15:36.832418 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.832378    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"]
Oct 30 22:15:36.833227 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.833206    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="a42b60f3-ab04-493b-aeab-ca862f753d56" containerName="cleanup"
Oct 30 22:15:36.836720 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.836699    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"
Oct 30 22:15:36.887927 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.887885    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbk5g\" (UniqueName: \"kubernetes.io/projected/3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce-kube-api-access-qbk5g\") pod \"payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682\" (UID: \"3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"
Oct 30 22:15:36.907318 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.907285    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"]
Oct 30 22:15:36.923961 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.923877    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w9rn\" (UniqueName: \"kubernetes.io/projected/0345916d-40ea-4739-985d-677b545547d7-kube-api-access-2w9rn\") pod \"payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km\" (UID: \"0345916d-40ea-4739-985d-677b545547d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"
Oct 30 22:15:36.924164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:36.924081    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0345916d-40ea-4739-985d-677b545547d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km\" (UID: \"0345916d-40ea-4739-985d-677b545547d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"
Oct 30 22:15:37.022380 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:37.022337    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"
Oct 30 22:15:37.024750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:37.024725    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0345916d-40ea-4739-985d-677b545547d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km\" (UID: \"0345916d-40ea-4739-985d-677b545547d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"
Oct 30 22:15:37.024844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:37.024831    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w9rn\" (UniqueName: \"kubernetes.io/projected/0345916d-40ea-4739-985d-677b545547d7-kube-api-access-2w9rn\") pod \"payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km\" (UID: \"0345916d-40ea-4739-985d-677b545547d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"
Oct 30 22:15:37.025742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:37.025498    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0345916d-40ea-4739-985d-677b545547d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km\" (UID: \"0345916d-40ea-4739-985d-677b545547d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"
Oct 30 22:15:37.084614 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:37.084181    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w9rn\" (UniqueName: \"kubernetes.io/projected/0345916d-40ea-4739-985d-677b545547d7-kube-api-access-2w9rn\") pod \"payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km\" (UID: \"0345916d-40ea-4739-985d-677b545547d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"
Oct 30 22:15:37.151964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:37.151921    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"
Oct 30 22:15:37.404312 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:37.404286    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"]
Oct 30 22:15:37.405280 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:15:37.405256    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d10f5a1_0ada_4aa7_b9d0_b0fc2ad958ce.slice/crio-e781af2b268ad8255a92c01c1aee2023fa0dd7b047c23eba1236316a9d7c248b WatchSource:0}: Error finding container e781af2b268ad8255a92c01c1aee2023fa0dd7b047c23eba1236316a9d7c248b: Status 404 returned error can't find the container with id e781af2b268ad8255a92c01c1aee2023fa0dd7b047c23eba1236316a9d7c248b
Oct 30 22:15:37.448175 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:37.448126    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682" event={"ID":"3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce","Type":"ContainerStarted","Data":"e781af2b268ad8255a92c01c1aee2023fa0dd7b047c23eba1236316a9d7c248b"}
Oct 30 22:15:37.537065 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:37.537041    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"]
Oct 30 22:15:37.538617 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:15:37.538589    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0345916d_40ea_4739_985d_677b545547d7.slice/crio-162ecfc661e682e7d3c28f274e12a0a9ebd5cc62ec6b93b0b6086432e573c398 WatchSource:0}: Error finding container 162ecfc661e682e7d3c28f274e12a0a9ebd5cc62ec6b93b0b6086432e573c398: Status 404 returned error can't find the container with id 162ecfc661e682e7d3c28f274e12a0a9ebd5cc62ec6b93b0b6086432e573c398
Oct 30 22:15:38.466768 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:38.466732    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km" event={"ID":"0345916d-40ea-4739-985d-677b545547d7","Type":"ContainerStarted","Data":"59eafa814c3ef6f4938d5277f88cb022f3da4cdab74908c693d255b04dc3617e"}
Oct 30 22:15:38.467794 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:38.466822    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"
Oct 30 22:15:38.467794 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:38.466847    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km" event={"ID":"0345916d-40ea-4739-985d-677b545547d7","Type":"ContainerStarted","Data":"162ecfc661e682e7d3c28f274e12a0a9ebd5cc62ec6b93b0b6086432e573c398"}
Oct 30 22:15:38.468509 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:38.468478    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682" event={"ID":"3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce","Type":"ContainerStarted","Data":"3054f3a576e876c09aa829c76d7c29993896c4c5b584c5de4cebeae8dfbdde40"}
Oct 30 22:15:38.468615 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:38.468540    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"
Oct 30 22:15:38.583459 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:38.583396    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km" podStartSLOduration=2.583378864 podStartE2EDuration="2.583378864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:38.582514277 +0000 UTC m=+1989.316877307" watchObservedRunningTime="2026-10-30 22:15:38.583378864 +0000 UTC m=+1989.317741895"
Oct 30 22:15:44.981454 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:44.981390    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682" podStartSLOduration=8.981373017 podStartE2EDuration="8.981373017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:38.709937322 +0000 UTC m=+1989.444300355" watchObservedRunningTime="2026-10-30 22:15:44.981373017 +0000 UTC m=+1995.715736053"
Oct 30 22:15:44.983504 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:44.983485    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"]
Oct 30 22:15:45.094793 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:45.094761    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"]
Oct 30 22:15:47.073979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.073705    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"]
Oct 30 22:15:47.093459 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.093414    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.112803 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.111150    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"]
Oct 30 22:15:47.118026 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.117100    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.147511 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.147485    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dz4rf\""
Oct 30 22:15:47.178642 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.178599    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.178642 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.178647    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.178910 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.178677    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.178910 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.178793    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.178910 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.178856    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.178910 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.178887    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.179066 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.178911    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.179066 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.178940    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvc5w\" (UniqueName: \"kubernetes.io/projected/a3a2bd94-6485-452d-98da-b656035143cb-kube-api-access-pvc5w\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.179066 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.178966    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.179066 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.179007    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.179318 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.179088    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bstql\" (UniqueName: \"kubernetes.io/projected/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-kube-api-access-bstql\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.179318 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.179115    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a2bd94-6485-452d-98da-b656035143cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.179318 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.179244    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.179318 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.179273    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a2bd94-6485-452d-98da-b656035143cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.179465 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.179338    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.179465 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.179376    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.179465 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.179404    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a2bd94-6485-452d-98da-b656035143cb-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.179569 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.179468    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.200838 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.200793    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"]
Oct 30 22:15:47.229474 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.229442    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"]
Oct 30 22:15:47.281034 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.280997    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.281250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281066    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bstql\" (UniqueName: \"kubernetes.io/projected/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-kube-api-access-bstql\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.281250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281086    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a2bd94-6485-452d-98da-b656035143cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.281250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281167    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.281250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281197    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a2bd94-6485-452d-98da-b656035143cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.281462 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281252    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.281462 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281290    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.281462 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281322    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a2bd94-6485-452d-98da-b656035143cb-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.281462 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281382    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.281462 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281446    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.281713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281478    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.281713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281502    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.281713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281513    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.281713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281506    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.281713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281577    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.281713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281626    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.281713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281655    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.281713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281683    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.282162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281717    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvc5w\" (UniqueName: \"kubernetes.io/projected/a3a2bd94-6485-452d-98da-b656035143cb-kube-api-access-pvc5w\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.282162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281728    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.282162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281754    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.282162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281757    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.282162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.281826    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.282454 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.282214    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a2bd94-6485-452d-98da-b656035143cb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.282454 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.282372    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.282662 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.282632    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.282746 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.282715    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.283170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.283108    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.284172 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.284106    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.284304 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.284288    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a2bd94-6485-452d-98da-b656035143cb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.284369 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.284348    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.284476 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.284461    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.341485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.341449    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.341651 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.341455    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bstql\" (UniqueName: \"kubernetes.io/projected/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-kube-api-access-bstql\") pod \"maas-default-gateway-openshift-default-694db55c87-pqpw6\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.341706 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.341642    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a2bd94-6485-452d-98da-b656035143cb-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.342002 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.341974    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvc5w\" (UniqueName: \"kubernetes.io/projected/a3a2bd94-6485-452d-98da-b656035143cb-kube-api-access-pvc5w\") pod \"maas-default-gateway-openshift-default-694db55c87-c8sl7\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.423762 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.423726    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:47.433170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.433124    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:47.860245 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.860211    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"]
Oct 30 22:15:47.861541 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:15:47.861516    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a2bd94_6485_452d_98da_b656035143cb.slice/crio-9bc5d016ed0c55ce15a86567f35e741d416e198a694839800cc51e50540871b8 WatchSource:0}: Error finding container 9bc5d016ed0c55ce15a86567f35e741d416e198a694839800cc51e50540871b8: Status 404 returned error can't find the container with id 9bc5d016ed0c55ce15a86567f35e741d416e198a694839800cc51e50540871b8
Oct 30 22:15:47.863160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.863088    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"]
Oct 30 22:15:47.864032 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.864001    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:15:47.864111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.864067    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:15:47.864111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.864094    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:15:47.904571 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:15:47.904528    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6dce00_fe43_484e_8d2f_3d1e6f480d53.slice/crio-e18800084acb85bd84884c26df7177d7e02987aa33e5628ae2210902f1f13973 WatchSource:0}: Error finding container e18800084acb85bd84884c26df7177d7e02987aa33e5628ae2210902f1f13973: Status 404 returned error can't find the container with id e18800084acb85bd84884c26df7177d7e02987aa33e5628ae2210902f1f13973
Oct 30 22:15:47.907427 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.907390    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:15:47.907567 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.907479    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:15:47.907567 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:47.907524    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:15:48.564170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:48.563338    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6" event={"ID":"ef6dce00-fe43-484e-8d2f-3d1e6f480d53","Type":"ContainerStarted","Data":"c20c412bda9d26913bc233736452221022b6efbbdaefed610e7f4c88652c8a45"}
Oct 30 22:15:48.564170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:48.563419    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6" event={"ID":"ef6dce00-fe43-484e-8d2f-3d1e6f480d53","Type":"ContainerStarted","Data":"e18800084acb85bd84884c26df7177d7e02987aa33e5628ae2210902f1f13973"}
Oct 30 22:15:48.569068 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:48.569026    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7" event={"ID":"a3a2bd94-6485-452d-98da-b656035143cb","Type":"ContainerStarted","Data":"9bfeaaec1b1967dc7bf0cba8305d2e3deee24e3936d140cd99496f58d60f07a2"}
Oct 30 22:15:48.569299 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:48.569120    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7" event={"ID":"a3a2bd94-6485-452d-98da-b656035143cb","Type":"ContainerStarted","Data":"9bc5d016ed0c55ce15a86567f35e741d416e198a694839800cc51e50540871b8"}
Oct 30 22:15:48.826225 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:48.819722    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6" podStartSLOduration=1.819699067 podStartE2EDuration="1.819699067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:48.691661966 +0000 UTC m=+1999.426024998" watchObservedRunningTime="2026-10-30 22:15:48.819699067 +0000 UTC m=+1999.554062099"
Oct 30 22:15:49.433771 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:49.433734    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:49.434030 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:49.434015    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:49.447161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:49.436247    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c8sl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:15:49.447161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:49.436337    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7" podUID="a3a2bd94-6485-452d-98da-b656035143cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 30 22:15:49.447161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:49.436665    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pqpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:15:49.447161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:49.436699    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6" podUID="ef6dce00-fe43-484e-8d2f-3d1e6f480d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 22:15:49.478872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:49.478839    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"
Oct 30 22:15:49.479331 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:49.479307    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"
Oct 30 22:15:49.598935 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:49.598868    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7" podStartSLOduration=2.59884934 podStartE2EDuration="2.59884934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:48.823875341 +0000 UTC m=+1999.558238373" watchObservedRunningTime="2026-10-30 22:15:49.59884934 +0000 UTC m=+2000.333212374"
Oct 30 22:15:49.983789 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:49.983742    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg" podUID="13383958-f736-4980-b167-e49792204928" containerName="payload-processing" containerID="cri-o://469bf7ed253515fdd1509cd60534946a7df027631d97f5772438ea41976b648f" gracePeriod=25
Oct 30 22:15:50.095665 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.095490    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9" podUID="189cad30-e568-4454-bf26-38fec456b0bd" containerName="payload-pre-processing" containerID="cri-o://bc386620a693458f36853d91bbd15c2dd7714979768f3f10118da0ed7bc94a5f" gracePeriod=25
Oct 30 22:15:50.463398 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.459046    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c8sl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:15:50.463398 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.459107    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7" podUID="a3a2bd94-6485-452d-98da-b656035143cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 30 22:15:50.463398 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.461037    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pqpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:15:50.463398 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.461080    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6" podUID="ef6dce00-fe43-484e-8d2f-3d1e6f480d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 22:15:50.471319 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.471289    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"
Oct 30 22:15:50.487916 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.487888    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13383958-f736-4980-b167-e49792204928-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13383958-f736-4980-b167-e49792204928-plugins-config-volume\") pod \"13383958-f736-4980-b167-e49792204928\" (UID: \"13383958-f736-4980-b167-e49792204928\") "
Oct 30 22:15:50.488057 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.487957    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13383958-f736-4980-b167-e49792204928-kube-api-access-5s2bh\" (UniqueName: \"kubernetes.io/projected/13383958-f736-4980-b167-e49792204928-kube-api-access-5s2bh\") pod \"13383958-f736-4980-b167-e49792204928\" (UID: \"13383958-f736-4980-b167-e49792204928\") "
Oct 30 22:15:50.488755 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.488735    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13383958-f736-4980-b167-e49792204928-plugins-config-volume" pod "13383958-f736-4980-b167-e49792204928" (UID: "13383958-f736-4980-b167-e49792204928"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:50.492570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.492018    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13383958-f736-4980-b167-e49792204928-kube-api-access-5s2bh" pod "13383958-f736-4980-b167-e49792204928" (UID: "13383958-f736-4980-b167-e49792204928"). InnerVolumeSpecName "kube-api-access-5s2bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:50.573345 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.573320    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"
Oct 30 22:15:50.585755 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.585722    4553 generic.go:356] "Generic (PLEG): container finished" podID="13383958-f736-4980-b167-e49792204928" containerID="469bf7ed253515fdd1509cd60534946a7df027631d97f5772438ea41976b648f" exitCode=0
Oct 30 22:15:50.585911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.585773    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"
Oct 30 22:15:50.585911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.585837    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg" event={"ID":"13383958-f736-4980-b167-e49792204928","Type":"ContainerDied","Data":"469bf7ed253515fdd1509cd60534946a7df027631d97f5772438ea41976b648f"}
Oct 30 22:15:50.585911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.585881    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg" event={"ID":"13383958-f736-4980-b167-e49792204928","Type":"ContainerDied","Data":"ff52f6a5d8f4e2c199e010044fc9fb628241b7a09256a9926f7b079ce2894439"}
Oct 30 22:15:50.585911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.585906    4553 scope.go:122] "RemoveContainer" containerID="469bf7ed253515fdd1509cd60534946a7df027631d97f5772438ea41976b648f"
Oct 30 22:15:50.589407 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.589379    4553 generic.go:356] "Generic (PLEG): container finished" podID="189cad30-e568-4454-bf26-38fec456b0bd" containerID="bc386620a693458f36853d91bbd15c2dd7714979768f3f10118da0ed7bc94a5f" exitCode=0
Oct 30 22:15:50.590437 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.590419    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"
Oct 30 22:15:50.590600 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.590581    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9" event={"ID":"189cad30-e568-4454-bf26-38fec456b0bd","Type":"ContainerDied","Data":"bc386620a693458f36853d91bbd15c2dd7714979768f3f10118da0ed7bc94a5f"}
Oct 30 22:15:50.590637 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.590626    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9" event={"ID":"189cad30-e568-4454-bf26-38fec456b0bd","Type":"ContainerDied","Data":"8741766a9ecdc1d5b7a3e419b31c1613ed44987ca62a67180a21a2baeed11127"}
Oct 30 22:15:50.591395 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.591374    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189cad30-e568-4454-bf26-38fec456b0bd-kube-api-access-ltmw4\" (UniqueName: \"kubernetes.io/projected/189cad30-e568-4454-bf26-38fec456b0bd-kube-api-access-ltmw4\") pod \"189cad30-e568-4454-bf26-38fec456b0bd\" (UID: \"189cad30-e568-4454-bf26-38fec456b0bd\") "
Oct 30 22:15:50.591578 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.591559    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189cad30-e568-4454-bf26-38fec456b0bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189cad30-e568-4454-bf26-38fec456b0bd-plugins-config-volume\") pod \"189cad30-e568-4454-bf26-38fec456b0bd\" (UID: \"189cad30-e568-4454-bf26-38fec456b0bd\") "
Oct 30 22:15:50.592120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.592105    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s2bh\" (UniqueName: \"kubernetes.io/projected/13383958-f736-4980-b167-e49792204928-kube-api-access-5s2bh\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:50.592267 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.592254    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13383958-f736-4980-b167-e49792204928-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:50.592331 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.592115    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189cad30-e568-4454-bf26-38fec456b0bd-plugins-config-volume" pod "189cad30-e568-4454-bf26-38fec456b0bd" (UID: "189cad30-e568-4454-bf26-38fec456b0bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:50.594192 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.594032    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189cad30-e568-4454-bf26-38fec456b0bd-kube-api-access-ltmw4" pod "189cad30-e568-4454-bf26-38fec456b0bd" (UID: "189cad30-e568-4454-bf26-38fec456b0bd"). InnerVolumeSpecName "kube-api-access-ltmw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:50.597974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.597959    4553 scope.go:122] "RemoveContainer" containerID="469bf7ed253515fdd1509cd60534946a7df027631d97f5772438ea41976b648f"
Oct 30 22:15:50.598352 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:15:50.598335    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"469bf7ed253515fdd1509cd60534946a7df027631d97f5772438ea41976b648f\": container with ID starting with 469bf7ed253515fdd1509cd60534946a7df027631d97f5772438ea41976b648f not found: ID does not exist" containerID="469bf7ed253515fdd1509cd60534946a7df027631d97f5772438ea41976b648f"
Oct 30 22:15:50.598412 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.598359    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469bf7ed253515fdd1509cd60534946a7df027631d97f5772438ea41976b648f"} err="failed to get container status \"469bf7ed253515fdd1509cd60534946a7df027631d97f5772438ea41976b648f\": rpc error: code = NotFound desc = could not find container \"469bf7ed253515fdd1509cd60534946a7df027631d97f5772438ea41976b648f\": container with ID starting with 469bf7ed253515fdd1509cd60534946a7df027631d97f5772438ea41976b648f not found: ID does not exist"
Oct 30 22:15:50.598412 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.598377    4553 scope.go:122] "RemoveContainer" containerID="bc386620a693458f36853d91bbd15c2dd7714979768f3f10118da0ed7bc94a5f"
Oct 30 22:15:50.608117 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.608080    4553 scope.go:122] "RemoveContainer" containerID="bc386620a693458f36853d91bbd15c2dd7714979768f3f10118da0ed7bc94a5f"
Oct 30 22:15:50.674157 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:15:50.672327    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc386620a693458f36853d91bbd15c2dd7714979768f3f10118da0ed7bc94a5f\": container with ID starting with bc386620a693458f36853d91bbd15c2dd7714979768f3f10118da0ed7bc94a5f not found: ID does not exist" containerID="bc386620a693458f36853d91bbd15c2dd7714979768f3f10118da0ed7bc94a5f"
Oct 30 22:15:50.674157 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.672380    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc386620a693458f36853d91bbd15c2dd7714979768f3f10118da0ed7bc94a5f"} err="failed to get container status \"bc386620a693458f36853d91bbd15c2dd7714979768f3f10118da0ed7bc94a5f\": rpc error: code = NotFound desc = could not find container \"bc386620a693458f36853d91bbd15c2dd7714979768f3f10118da0ed7bc94a5f\": container with ID starting with bc386620a693458f36853d91bbd15c2dd7714979768f3f10118da0ed7bc94a5f not found: ID does not exist"
Oct 30 22:15:50.694190 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.694124    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189cad30-e568-4454-bf26-38fec456b0bd-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:50.694190 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.694182    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltmw4\" (UniqueName: \"kubernetes.io/projected/189cad30-e568-4454-bf26-38fec456b0bd-kube-api-access-ltmw4\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:50.966413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.966296    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"]
Oct 30 22:15:50.969241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:50.969220    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-75ff567cf9-jwtbg"]
Oct 30 22:15:51.141163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:51.140194    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"]
Oct 30 22:15:51.162008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:51.161906    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-56b77c4f9f-2zct9"]
Oct 30 22:15:51.395111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:51.395078    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13383958-f736-4980-b167-e49792204928" path="/var/lib/kubelet/pods/13383958-f736-4980-b167-e49792204928/volumes"
Oct 30 22:15:51.395594 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:51.395578    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189cad30-e568-4454-bf26-38fec456b0bd" path="/var/lib/kubelet/pods/189cad30-e568-4454-bf26-38fec456b0bd/volumes"
Oct 30 22:15:51.428225 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:51.425571    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c8sl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:15:51.428225 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:51.425648    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7" podUID="a3a2bd94-6485-452d-98da-b656035143cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 30 22:15:51.444165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:51.435574    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pqpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:15:51.444165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:51.435634    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6" podUID="ef6dce00-fe43-484e-8d2f-3d1e6f480d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 22:15:52.425162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:52.424574    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c8sl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:15:52.425162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:52.424648    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7" podUID="a3a2bd94-6485-452d-98da-b656035143cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 30 22:15:52.448165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:52.444565    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pqpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:15:52.448165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:52.444624    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6" podUID="ef6dce00-fe43-484e-8d2f-3d1e6f480d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 22:15:53.425160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:53.425003    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c8sl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:15:53.425160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:53.425071    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7" podUID="a3a2bd94-6485-452d-98da-b656035143cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 30 22:15:53.441163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:53.437591    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pqpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:15:53.441163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:53.437656    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6" podUID="ef6dce00-fe43-484e-8d2f-3d1e6f480d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 22:15:54.425053 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:54.425011    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c8sl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:15:54.425254 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:54.425098    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7" podUID="a3a2bd94-6485-452d-98da-b656035143cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 30 22:15:54.434456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:54.434431    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pqpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:15:54.434821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:54.434483    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6" podUID="ef6dce00-fe43-484e-8d2f-3d1e6f480d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 22:15:55.424397 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:55.424353    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c8sl7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 30 22:15:55.424607 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:55.424428    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7" podUID="a3a2bd94-6485-452d-98da-b656035143cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 30 22:15:55.434379 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:55.434350    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pqpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:15:55.434540 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:55.434403    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6" podUID="ef6dce00-fe43-484e-8d2f-3d1e6f480d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 22:15:56.677191 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:56.676727    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:56.678192 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:56.678168    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:56.678857 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:56.678835    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:15:56.724533 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:56.724496    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:56.724950 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:56.724922    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:56.725763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:15:56.725745    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:15:58.202490 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:15:58.202445    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:16:02.212701 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.212665    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"]
Oct 30 22:16:02.213362 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.213346    4553 cpu_manager.go:414] "containerMap: removing container" podUID="13383958-f736-4980-b167-e49792204928" containerName="payload-processing"
Oct 30 22:16:02.213429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.213363    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13383958-f736-4980-b167-e49792204928" containerName="payload-processing"
Oct 30 22:16:02.213429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.213381    4553 cpu_manager.go:414] "containerMap: removing container" podUID="189cad30-e568-4454-bf26-38fec456b0bd" containerName="payload-pre-processing"
Oct 30 22:16:02.213429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.213386    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189cad30-e568-4454-bf26-38fec456b0bd" containerName="payload-pre-processing"
Oct 30 22:16:02.213550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.213482    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="189cad30-e568-4454-bf26-38fec456b0bd" containerName="payload-pre-processing"
Oct 30 22:16:02.213550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.213500    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="13383958-f736-4980-b167-e49792204928" containerName="payload-processing"
Oct 30 22:16:02.218385 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.218363    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.294357 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.294320    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5tr7\" (UniqueName: \"kubernetes.io/projected/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-kube-api-access-k5tr7\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.294567 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.294381    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.294567 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.294487    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.294567 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.294531    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.294706 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.294591    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.294754 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.294739    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.294796 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.294769    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.294796 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.294788    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.294907 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.294842    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.396549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.396509    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.396549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.396551    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.396821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.396596    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.396821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.396723    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.396821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.396750    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.396821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.396776    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.397028 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.396841    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.397192 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.397125    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.397260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.397186    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.397320 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.397296    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.397563 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.397536    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.397693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.397635    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5tr7\" (UniqueName: \"kubernetes.io/projected/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-kube-api-access-k5tr7\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.397758 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.397695    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.397993 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.397974    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.399058 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.399031    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.399387 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.399369    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.421671 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.421628    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"]
Oct 30 22:16:02.455067 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.455030    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.455872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.455827    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5tr7\" (UniqueName: \"kubernetes.io/projected/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-kube-api-access-k5tr7\") pod \"maas-default-gateway-openshift-default-694db55c87-bxr9x\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.533987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.533890    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:02.965240 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.965212    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"]
Oct 30 22:16:02.966437 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:16:02.966400    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1dc2e21_d5fb_4dc6_895d_fb651fa2ed90.slice/crio-72f0b20d702fc6e7a2671bd98fc5d65c339dd66ec5278ea5eb9222e43004b238 WatchSource:0}: Error finding container 72f0b20d702fc6e7a2671bd98fc5d65c339dd66ec5278ea5eb9222e43004b238: Status 404 returned error can't find the container with id 72f0b20d702fc6e7a2671bd98fc5d65c339dd66ec5278ea5eb9222e43004b238
Oct 30 22:16:02.969037 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.969005    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:16:02.969114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.969079    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:16:02.969114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:02.969106    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:16:03.683124 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:03.683085    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x" event={"ID":"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90","Type":"ContainerStarted","Data":"40dc8616ea9fdfb5c82376fe70b5f386111fd57317e5a35fa7875f3b37787045"}
Oct 30 22:16:03.683606 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:03.683189    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x" event={"ID":"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90","Type":"ContainerStarted","Data":"72f0b20d702fc6e7a2671bd98fc5d65c339dd66ec5278ea5eb9222e43004b238"}
Oct 30 22:16:03.853694 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:03.853640    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x" podStartSLOduration=1.853623884 podStartE2EDuration="1.853623884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:03.851205598 +0000 UTC m=+2014.585568609" watchObservedRunningTime="2026-10-30 22:16:03.853623884 +0000 UTC m=+2014.587987008"
Oct 30 22:16:04.534639 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:04.534599    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:04.536409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:04.536377    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bxr9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:16:04.536558 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:04.536440    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x" podUID="f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 22:16:05.534412 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:05.534374    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bxr9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:16:05.534827 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:05.534444    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x" podUID="f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 22:16:06.535489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:06.535444    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bxr9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:16:06.535916 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:06.535518    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x" podUID="f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 22:16:07.534726 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:07.534679    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bxr9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:16:07.534952 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:07.534759    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x" podUID="f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 22:16:08.534741 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:08.534698    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bxr9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:16:08.535203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:08.534802    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x" podUID="f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 22:16:09.534997 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:09.534915    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bxr9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:16:09.534997 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:09.534970    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x" podUID="f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 22:16:10.538363 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:10.538322    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bxr9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:16:10.539272 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:10.539236    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x" podUID="f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 30 22:16:11.766769 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:11.766731    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:11.767858 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:11.767317    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:11.768664 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:11.768643    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:16:23.188748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:23.188713    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"]
Oct 30 22:16:23.193008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:23.192982    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"
Oct 30 22:16:23.219744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:23.219713    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g54db\" (UniqueName: \"kubernetes.io/projected/4be56767-e095-44c3-9a5b-5952102959cb-kube-api-access-g54db\") pod \"payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf\" (UID: \"4be56767-e095-44c3-9a5b-5952102959cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"
Oct 30 22:16:23.219921 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:23.219896    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be56767-e095-44c3-9a5b-5952102959cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf\" (UID: \"4be56767-e095-44c3-9a5b-5952102959cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"
Oct 30 22:16:23.248622 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:23.248589    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"]
Oct 30 22:16:23.295750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:23.295715    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7a54db-dockercfg-gvfxw\""
Oct 30 22:16:23.295750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:23.295736    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7a54db\""
Oct 30 22:16:23.321468 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:23.321431    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g54db\" (UniqueName: \"kubernetes.io/projected/4be56767-e095-44c3-9a5b-5952102959cb-kube-api-access-g54db\") pod \"payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf\" (UID: \"4be56767-e095-44c3-9a5b-5952102959cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"
Oct 30 22:16:23.321641 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:23.321555    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be56767-e095-44c3-9a5b-5952102959cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf\" (UID: \"4be56767-e095-44c3-9a5b-5952102959cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"
Oct 30 22:16:23.322290 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:23.322273    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be56767-e095-44c3-9a5b-5952102959cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf\" (UID: \"4be56767-e095-44c3-9a5b-5952102959cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"
Oct 30 22:16:23.379589 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:23.379550    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g54db\" (UniqueName: \"kubernetes.io/projected/4be56767-e095-44c3-9a5b-5952102959cb-kube-api-access-g54db\") pod \"payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf\" (UID: \"4be56767-e095-44c3-9a5b-5952102959cb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"
Oct 30 22:16:23.508384 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:23.508291    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"
Oct 30 22:16:23.886626 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:23.886592    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"]
Oct 30 22:16:23.886784 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:16:23.886649    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be56767_e095_44c3_9a5b_5952102959cb.slice/crio-f3661c4aae0d278146e4af819378fdabd401a9f34f1d59918ef8e68c9615dee6 WatchSource:0}: Error finding container f3661c4aae0d278146e4af819378fdabd401a9f34f1d59918ef8e68c9615dee6: Status 404 returned error can't find the container with id f3661c4aae0d278146e4af819378fdabd401a9f34f1d59918ef8e68c9615dee6
Oct 30 22:16:23.974768 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:23.974733    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"]
Oct 30 22:16:23.979436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:23.979413    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"
Oct 30 22:16:24.032346 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.032297    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x\" (UID: \"d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"
Oct 30 22:16:24.058421 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.058296    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqnpc\" (UniqueName: \"kubernetes.io/projected/d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc-kube-api-access-mqnpc\") pod \"payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x\" (UID: \"d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"
Oct 30 22:16:24.059760 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.042974    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"]
Oct 30 22:16:24.167180 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.167059    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x\" (UID: \"d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"
Oct 30 22:16:24.167180 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.167173    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqnpc\" (UniqueName: \"kubernetes.io/projected/d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc-kube-api-access-mqnpc\") pod \"payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x\" (UID: \"d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"
Oct 30 22:16:24.168506 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.168479    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x\" (UID: \"d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"
Oct 30 22:16:24.231126 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.228452    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqnpc\" (UniqueName: \"kubernetes.io/projected/d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc-kube-api-access-mqnpc\") pod \"payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x\" (UID: \"d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"
Oct 30 22:16:24.301157 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.295560    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"
Oct 30 22:16:24.529329 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.529233    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"]
Oct 30 22:16:24.529511 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.529487    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww" podUID="5b2c8fd1-3f67-423d-92a7-40923ce4848a" containerName="payload-processing" containerID="cri-o://ce06f58525bc48c640a4a23ac7b97c088b2f98d3d95facc165ee66500306ded9" gracePeriod=30
Oct 30 22:16:24.580831 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.580801    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"]
Oct 30 22:16:24.581016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.580997    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692" podUID="41752921-dba6-4eec-ac05-1971cba00908" containerName="payload-pre-processing" containerID="cri-o://fca8dc48e6576317abd3611f850a669608ec3c59a52e0cbcfb68050c9bcb1152" gracePeriod=30
Oct 30 22:16:24.682597 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.682564    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"]
Oct 30 22:16:24.683683 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:16:24.683654    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d40c35_7fbd_4b56_9bd4_1f45b0b198cc.slice/crio-7e74005671113a6c5ebc9f656acf2f329202f801daf0071abffdcb3aaefe870f WatchSource:0}: Error finding container 7e74005671113a6c5ebc9f656acf2f329202f801daf0071abffdcb3aaefe870f: Status 404 returned error can't find the container with id 7e74005671113a6c5ebc9f656acf2f329202f801daf0071abffdcb3aaefe870f
Oct 30 22:16:24.875330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.875299    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"
Oct 30 22:16:24.893747 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.893709    4553 generic.go:356] "Generic (PLEG): container finished" podID="41752921-dba6-4eec-ac05-1971cba00908" containerID="fca8dc48e6576317abd3611f850a669608ec3c59a52e0cbcfb68050c9bcb1152" exitCode=0
Oct 30 22:16:24.893925 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.893810    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692" event={"ID":"41752921-dba6-4eec-ac05-1971cba00908","Type":"ContainerDied","Data":"fca8dc48e6576317abd3611f850a669608ec3c59a52e0cbcfb68050c9bcb1152"}
Oct 30 22:16:24.895559 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.895522    4553 generic.go:356] "Generic (PLEG): container finished" podID="5b2c8fd1-3f67-423d-92a7-40923ce4848a" containerID="ce06f58525bc48c640a4a23ac7b97c088b2f98d3d95facc165ee66500306ded9" exitCode=0
Oct 30 22:16:24.895697 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.895568    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"
Oct 30 22:16:24.895697 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.895628    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww" event={"ID":"5b2c8fd1-3f67-423d-92a7-40923ce4848a","Type":"ContainerDied","Data":"ce06f58525bc48c640a4a23ac7b97c088b2f98d3d95facc165ee66500306ded9"}
Oct 30 22:16:24.895697 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.895670    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww" event={"ID":"5b2c8fd1-3f67-423d-92a7-40923ce4848a","Type":"ContainerDied","Data":"57bcca9d7aff97f5a9d9c16fdc5b99ff45c0cf15e1a85616c9d85f04c1eb02a2"}
Oct 30 22:16:24.895849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.895698    4553 scope.go:122] "RemoveContainer" containerID="ce06f58525bc48c640a4a23ac7b97c088b2f98d3d95facc165ee66500306ded9"
Oct 30 22:16:24.898819 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.898091    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x" event={"ID":"d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc","Type":"ContainerStarted","Data":"afe3cbf5acc746413616083699d5e207b7f7a816df902afc11f9f42caec90f3a"}
Oct 30 22:16:24.898819 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.898181    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x" event={"ID":"d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc","Type":"ContainerStarted","Data":"7e74005671113a6c5ebc9f656acf2f329202f801daf0071abffdcb3aaefe870f"}
Oct 30 22:16:24.898819 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.898208    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"
Oct 30 22:16:24.900937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.900912    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf" event={"ID":"4be56767-e095-44c3-9a5b-5952102959cb","Type":"ContainerStarted","Data":"d0cff845328c9427daeabb0283305ee9292ab99a70cb28781c903aebec8be910"}
Oct 30 22:16:24.901048 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.900960    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf" event={"ID":"4be56767-e095-44c3-9a5b-5952102959cb","Type":"ContainerStarted","Data":"f3661c4aae0d278146e4af819378fdabd401a9f34f1d59918ef8e68c9615dee6"}
Oct 30 22:16:24.901048 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.900989    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"
Oct 30 22:16:24.908843 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.908821    4553 scope.go:122] "RemoveContainer" containerID="ce06f58525bc48c640a4a23ac7b97c088b2f98d3d95facc165ee66500306ded9"
Oct 30 22:16:24.910302 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:16:24.910275    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce06f58525bc48c640a4a23ac7b97c088b2f98d3d95facc165ee66500306ded9\": container with ID starting with ce06f58525bc48c640a4a23ac7b97c088b2f98d3d95facc165ee66500306ded9 not found: ID does not exist" containerID="ce06f58525bc48c640a4a23ac7b97c088b2f98d3d95facc165ee66500306ded9"
Oct 30 22:16:24.910409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.910316    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce06f58525bc48c640a4a23ac7b97c088b2f98d3d95facc165ee66500306ded9"} err="failed to get container status \"ce06f58525bc48c640a4a23ac7b97c088b2f98d3d95facc165ee66500306ded9\": rpc error: code = NotFound desc = could not find container \"ce06f58525bc48c640a4a23ac7b97c088b2f98d3d95facc165ee66500306ded9\": container with ID starting with ce06f58525bc48c640a4a23ac7b97c088b2f98d3d95facc165ee66500306ded9 not found: ID does not exist"
Oct 30 22:16:24.929862 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.929839    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"
Oct 30 22:16:24.983561 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.983527    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b2c8fd1-3f67-423d-92a7-40923ce4848a-kube-api-access-l4sjm\" (UniqueName: \"kubernetes.io/projected/5b2c8fd1-3f67-423d-92a7-40923ce4848a-kube-api-access-l4sjm\") pod \"5b2c8fd1-3f67-423d-92a7-40923ce4848a\" (UID: \"5b2c8fd1-3f67-423d-92a7-40923ce4848a\") "
Oct 30 22:16:24.983561 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.983567    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b2c8fd1-3f67-423d-92a7-40923ce4848a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2c8fd1-3f67-423d-92a7-40923ce4848a-plugins-config-volume\") pod \"5b2c8fd1-3f67-423d-92a7-40923ce4848a\" (UID: \"5b2c8fd1-3f67-423d-92a7-40923ce4848a\") "
Oct 30 22:16:24.983785 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.983608    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41752921-dba6-4eec-ac05-1971cba00908-kube-api-access-vvw95\" (UniqueName: \"kubernetes.io/projected/41752921-dba6-4eec-ac05-1971cba00908-kube-api-access-vvw95\") pod \"41752921-dba6-4eec-ac05-1971cba00908\" (UID: \"41752921-dba6-4eec-ac05-1971cba00908\") "
Oct 30 22:16:24.983785 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.983638    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41752921-dba6-4eec-ac05-1971cba00908-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41752921-dba6-4eec-ac05-1971cba00908-plugins-config-volume\") pod \"41752921-dba6-4eec-ac05-1971cba00908\" (UID: \"41752921-dba6-4eec-ac05-1971cba00908\") "
Oct 30 22:16:24.984051 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.984023    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b2c8fd1-3f67-423d-92a7-40923ce4848a-plugins-config-volume" pod "5b2c8fd1-3f67-423d-92a7-40923ce4848a" (UID: "5b2c8fd1-3f67-423d-92a7-40923ce4848a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:24.984197 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.984095    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41752921-dba6-4eec-ac05-1971cba00908-plugins-config-volume" pod "41752921-dba6-4eec-ac05-1971cba00908" (UID: "41752921-dba6-4eec-ac05-1971cba00908"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:24.984356 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.984333    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b2c8fd1-3f67-423d-92a7-40923ce4848a-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:24.984416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.984356    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41752921-dba6-4eec-ac05-1971cba00908-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:24.985960 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.985937    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b2c8fd1-3f67-423d-92a7-40923ce4848a-kube-api-access-l4sjm" pod "5b2c8fd1-3f67-423d-92a7-40923ce4848a" (UID: "5b2c8fd1-3f67-423d-92a7-40923ce4848a"). InnerVolumeSpecName "kube-api-access-l4sjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:24.985960 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:24.985955    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41752921-dba6-4eec-ac05-1971cba00908-kube-api-access-vvw95" pod "41752921-dba6-4eec-ac05-1971cba00908" (UID: "41752921-dba6-4eec-ac05-1971cba00908"). InnerVolumeSpecName "kube-api-access-vvw95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:25.095035 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:25.095001    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4sjm\" (UniqueName: \"kubernetes.io/projected/5b2c8fd1-3f67-423d-92a7-40923ce4848a-kube-api-access-l4sjm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:25.095321 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:25.095302    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvw95\" (UniqueName: \"kubernetes.io/projected/41752921-dba6-4eec-ac05-1971cba00908-kube-api-access-vvw95\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:25.180196 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:25.175181    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf" podStartSLOduration=2.175159823 podStartE2EDuration="2.175159823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:25.173871454 +0000 UTC m=+2035.908234486" watchObservedRunningTime="2026-10-30 22:16:25.175159823 +0000 UTC m=+2035.909522856"
Oct 30 22:16:25.294503 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:25.294442    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x" podStartSLOduration=2.2944278750000002 podStartE2EDuration="2.294427875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:25.291274395 +0000 UTC m=+2036.025637427" watchObservedRunningTime="2026-10-30 22:16:25.294427875 +0000 UTC m=+2036.028790918"
Oct 30 22:16:25.595586 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:25.595547    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"]
Oct 30 22:16:25.599643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:25.599612    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97ca39-7dfd9c7df8-5dpww"]
Oct 30 22:16:25.908695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:25.908662    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"
Oct 30 22:16:25.908877 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:25.908658    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692" event={"ID":"41752921-dba6-4eec-ac05-1971cba00908","Type":"ContainerDied","Data":"115e9f1343a16a24fe4a459bbc645df7d1ada7db63d41703e61d6124d713e170"}
Oct 30 22:16:25.908877 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:25.908804    4553 scope.go:122] "RemoveContainer" containerID="fca8dc48e6576317abd3611f850a669608ec3c59a52e0cbcfb68050c9bcb1152"
Oct 30 22:16:26.076057 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:26.076024    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"]
Oct 30 22:16:26.085931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:26.085891    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97ca39-574f455b47-cb692"]
Oct 30 22:16:27.393918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:27.393885    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41752921-dba6-4eec-ac05-1971cba00908" path="/var/lib/kubelet/pods/41752921-dba6-4eec-ac05-1971cba00908/volumes"
Oct 30 22:16:27.394397 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:27.394381    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b2c8fd1-3f67-423d-92a7-40923ce4848a" path="/var/lib/kubelet/pods/5b2c8fd1-3f67-423d-92a7-40923ce4848a/volumes"
Oct 30 22:16:28.202639 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:16:28.202604    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:16:30.054242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:30.054208    4553 scope.go:122] "RemoveContainer" containerID="aee38380a5d5d5227fa5d405651d0b1d73f9aef6732031aafc4cfd8f6c7dfb30"
Oct 30 22:16:35.913549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:35.913519    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"
Oct 30 22:16:35.913987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:35.913595    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"
Oct 30 22:16:41.619074 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.619038    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"]
Oct 30 22:16:41.619682 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.619666    4553 cpu_manager.go:414] "containerMap: removing container" podUID="41752921-dba6-4eec-ac05-1971cba00908" containerName="payload-pre-processing"
Oct 30 22:16:41.619733 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.619683    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41752921-dba6-4eec-ac05-1971cba00908" containerName="payload-pre-processing"
Oct 30 22:16:41.619733 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.619712    4553 cpu_manager.go:414] "containerMap: removing container" podUID="5b2c8fd1-3f67-423d-92a7-40923ce4848a" containerName="payload-processing"
Oct 30 22:16:41.619733 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.619717    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b2c8fd1-3f67-423d-92a7-40923ce4848a" containerName="payload-processing"
Oct 30 22:16:41.619831 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.619799    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="41752921-dba6-4eec-ac05-1971cba00908" containerName="payload-pre-processing"
Oct 30 22:16:41.619831 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.619812    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b2c8fd1-3f67-423d-92a7-40923ce4848a" containerName="payload-processing"
Oct 30 22:16:41.625518 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.625498    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"
Oct 30 22:16:41.683445 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.683415    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"]
Oct 30 22:16:41.730298 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.730263    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-69a1d2\""
Oct 30 22:16:41.730298 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.730274    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-69a1d2-dockercfg-gdm9g\""
Oct 30 22:16:41.756804 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.756762    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp5tl\" (UniqueName: \"kubernetes.io/projected/1dd04861-5971-4a70-9a64-ce27e2f9f6b2-kube-api-access-pp5tl\") pod \"payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k\" (UID: \"1dd04861-5971-4a70-9a64-ce27e2f9f6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"
Oct 30 22:16:41.757004 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.756980    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd04861-5971-4a70-9a64-ce27e2f9f6b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k\" (UID: \"1dd04861-5971-4a70-9a64-ce27e2f9f6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"
Oct 30 22:16:41.858803 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.858771    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd04861-5971-4a70-9a64-ce27e2f9f6b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k\" (UID: \"1dd04861-5971-4a70-9a64-ce27e2f9f6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"
Oct 30 22:16:41.858987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.858927    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp5tl\" (UniqueName: \"kubernetes.io/projected/1dd04861-5971-4a70-9a64-ce27e2f9f6b2-kube-api-access-pp5tl\") pod \"payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k\" (UID: \"1dd04861-5971-4a70-9a64-ce27e2f9f6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"
Oct 30 22:16:41.859564 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.859543    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd04861-5971-4a70-9a64-ce27e2f9f6b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k\" (UID: \"1dd04861-5971-4a70-9a64-ce27e2f9f6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"
Oct 30 22:16:41.917050 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.916977    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp5tl\" (UniqueName: \"kubernetes.io/projected/1dd04861-5971-4a70-9a64-ce27e2f9f6b2-kube-api-access-pp5tl\") pod \"payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k\" (UID: \"1dd04861-5971-4a70-9a64-ce27e2f9f6b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"
Oct 30 22:16:41.939544 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:41.939513    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"
Oct 30 22:16:42.309446 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:42.309367    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"]
Oct 30 22:16:42.315308 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:42.315277    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"
Oct 30 22:16:42.319483 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:16:42.319459    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd04861_5971_4a70_9a64_ce27e2f9f6b2.slice/crio-94d8502d12a35ddf84b822826d9a1d495c0d2f263b075bea261dbd1e234bc746 WatchSource:0}: Error finding container 94d8502d12a35ddf84b822826d9a1d495c0d2f263b075bea261dbd1e234bc746: Status 404 returned error can't find the container with id 94d8502d12a35ddf84b822826d9a1d495c0d2f263b075bea261dbd1e234bc746
Oct 30 22:16:42.319729 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:42.319708    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"]
Oct 30 22:16:42.370643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:42.370617    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"]
Oct 30 22:16:42.467479 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:42.467442    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47b5g\" (UniqueName: \"kubernetes.io/projected/5b89e92c-e8c0-4282-b6e5-a4c8f867e08b-kube-api-access-47b5g\") pod \"payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp\" (UID: \"5b89e92c-e8c0-4282-b6e5-a4c8f867e08b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"
Oct 30 22:16:42.467678 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:42.467634    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b89e92c-e8c0-4282-b6e5-a4c8f867e08b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp\" (UID: \"5b89e92c-e8c0-4282-b6e5-a4c8f867e08b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"
Oct 30 22:16:42.568753 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:42.568664    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b89e92c-e8c0-4282-b6e5-a4c8f867e08b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp\" (UID: \"5b89e92c-e8c0-4282-b6e5-a4c8f867e08b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"
Oct 30 22:16:42.568933 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:42.568780    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47b5g\" (UniqueName: \"kubernetes.io/projected/5b89e92c-e8c0-4282-b6e5-a4c8f867e08b-kube-api-access-47b5g\") pod \"payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp\" (UID: \"5b89e92c-e8c0-4282-b6e5-a4c8f867e08b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"
Oct 30 22:16:42.569469 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:42.569444    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b89e92c-e8c0-4282-b6e5-a4c8f867e08b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp\" (UID: \"5b89e92c-e8c0-4282-b6e5-a4c8f867e08b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"
Oct 30 22:16:42.627338 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:42.627308    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47b5g\" (UniqueName: \"kubernetes.io/projected/5b89e92c-e8c0-4282-b6e5-a4c8f867e08b-kube-api-access-47b5g\") pod \"payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp\" (UID: \"5b89e92c-e8c0-4282-b6e5-a4c8f867e08b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"
Oct 30 22:16:42.666512 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:42.666482    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"
Oct 30 22:16:43.043523 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:43.043484    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k" event={"ID":"1dd04861-5971-4a70-9a64-ce27e2f9f6b2","Type":"ContainerStarted","Data":"b437a265d2aa5c89b69970b26f7685a9050fbb58aa163ccb4885c1911b5b09ad"}
Oct 30 22:16:43.043710 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:43.043587    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"
Oct 30 22:16:43.043710 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:43.043609    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k" event={"ID":"1dd04861-5971-4a70-9a64-ce27e2f9f6b2","Type":"ContainerStarted","Data":"94d8502d12a35ddf84b822826d9a1d495c0d2f263b075bea261dbd1e234bc746"}
Oct 30 22:16:43.058577 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:43.058552    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"]
Oct 30 22:16:43.059437 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:16:43.059412    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b89e92c_e8c0_4282_b6e5_a4c8f867e08b.slice/crio-6e05abf42524b7274826d847fcd81ebb0bc703eb447f0392e6660d591f732bbf WatchSource:0}: Error finding container 6e05abf42524b7274826d847fcd81ebb0bc703eb447f0392e6660d591f732bbf: Status 404 returned error can't find the container with id 6e05abf42524b7274826d847fcd81ebb0bc703eb447f0392e6660d591f732bbf
Oct 30 22:16:43.160956 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:43.160900    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k" podStartSLOduration=2.160879854 podStartE2EDuration="2.160879854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:43.159505665 +0000 UTC m=+2053.893868699" watchObservedRunningTime="2026-10-30 22:16:43.160879854 +0000 UTC m=+2053.895242886"
Oct 30 22:16:44.053094 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:44.053054    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp" event={"ID":"5b89e92c-e8c0-4282-b6e5-a4c8f867e08b","Type":"ContainerStarted","Data":"e98a167502aebf771a562f0ca0635da95b7b854718dd292f9f86ac56f989ce9c"}
Oct 30 22:16:44.053586 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:44.053114    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp" event={"ID":"5b89e92c-e8c0-4282-b6e5-a4c8f867e08b","Type":"ContainerStarted","Data":"6e05abf42524b7274826d847fcd81ebb0bc703eb447f0392e6660d591f732bbf"}
Oct 30 22:16:44.053586 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:44.053159    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"
Oct 30 22:16:44.172464 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:44.172412    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp" podStartSLOduration=2.172396754 podStartE2EDuration="2.172396754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:44.170377327 +0000 UTC m=+2054.904740382" watchObservedRunningTime="2026-10-30 22:16:44.172396754 +0000 UTC m=+2054.906759786"
Oct 30 22:16:52.644347 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:52.644302    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"]
Oct 30 22:16:52.644884 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:52.644653    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl" podUID="ebef8f6b-459e-4a01-8fa2-ed06d04208c5" containerName="maas-api" containerID="cri-o://8572f73e012c549a0492c89983b5afa7d94a43c51ab93be7d2a15025051db5c3" gracePeriod=30
Oct 30 22:16:52.980023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:52.979999    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:16:53.056166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.047673    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-kube-api-access-w7nxb\" (UniqueName: \"kubernetes.io/projected/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-kube-api-access-w7nxb\") pod \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") "
Oct 30 22:16:53.056166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.047766    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-maas-api-tls\") pod \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") "
Oct 30 22:16:53.056166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.047875    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-openshift-service-ca\") pod \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") "
Oct 30 22:16:53.056166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.047999    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-metrics-certs\") pod \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\" (UID: \"ebef8f6b-459e-4a01-8fa2-ed06d04208c5\") "
Oct 30 22:16:53.078160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.077707    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-openshift-service-ca" pod "ebef8f6b-459e-4a01-8fa2-ed06d04208c5" (UID: "ebef8f6b-459e-4a01-8fa2-ed06d04208c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:53.095391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.095335    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-kube-api-access-w7nxb" pod "ebef8f6b-459e-4a01-8fa2-ed06d04208c5" (UID: "ebef8f6b-459e-4a01-8fa2-ed06d04208c5"). InnerVolumeSpecName "kube-api-access-w7nxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:53.095590 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.095566    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-metrics-certs" pod "ebef8f6b-459e-4a01-8fa2-ed06d04208c5" (UID: "ebef8f6b-459e-4a01-8fa2-ed06d04208c5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:53.095674 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.095659    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-maas-api-tls" pod "ebef8f6b-459e-4a01-8fa2-ed06d04208c5" (UID: "ebef8f6b-459e-4a01-8fa2-ed06d04208c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:53.123931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.123893    4553 generic.go:356] "Generic (PLEG): container finished" podID="ebef8f6b-459e-4a01-8fa2-ed06d04208c5" containerID="8572f73e012c549a0492c89983b5afa7d94a43c51ab93be7d2a15025051db5c3" exitCode=0
Oct 30 22:16:53.124115 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.123955    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"
Oct 30 22:16:53.124115 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.123962    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl" event={"ID":"ebef8f6b-459e-4a01-8fa2-ed06d04208c5","Type":"ContainerDied","Data":"8572f73e012c549a0492c89983b5afa7d94a43c51ab93be7d2a15025051db5c3"}
Oct 30 22:16:53.124115 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.124055    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl" event={"ID":"ebef8f6b-459e-4a01-8fa2-ed06d04208c5","Type":"ContainerDied","Data":"4006e1c005f4c5f45ce051de2782cd23793635aa2137e4a51d4b76b2f784ad81"}
Oct 30 22:16:53.124115 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.124090    4553 scope.go:122] "RemoveContainer" containerID="8572f73e012c549a0492c89983b5afa7d94a43c51ab93be7d2a15025051db5c3"
Oct 30 22:16:53.124456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.124435    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"]
Oct 30 22:16:53.124621 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.124600    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr" podUID="b277a651-334b-4560-9dc9-89bf75a1b9a6" containerName="payload-processing" containerID="cri-o://f2ed814b5f12dec9bdbe496de1d3148f0df05a1a8438acb91a8cc313df388bad" gracePeriod=30
Oct 30 22:16:53.140054 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.139956    4553 scope.go:122] "RemoveContainer" containerID="8572f73e012c549a0492c89983b5afa7d94a43c51ab93be7d2a15025051db5c3"
Oct 30 22:16:53.140385 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:16:53.140362    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8572f73e012c549a0492c89983b5afa7d94a43c51ab93be7d2a15025051db5c3\": container with ID starting with 8572f73e012c549a0492c89983b5afa7d94a43c51ab93be7d2a15025051db5c3 not found: ID does not exist" containerID="8572f73e012c549a0492c89983b5afa7d94a43c51ab93be7d2a15025051db5c3"
Oct 30 22:16:53.140454 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.140396    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8572f73e012c549a0492c89983b5afa7d94a43c51ab93be7d2a15025051db5c3"} err="failed to get container status \"8572f73e012c549a0492c89983b5afa7d94a43c51ab93be7d2a15025051db5c3\": rpc error: code = NotFound desc = could not find container \"8572f73e012c549a0492c89983b5afa7d94a43c51ab93be7d2a15025051db5c3\": container with ID starting with 8572f73e012c549a0492c89983b5afa7d94a43c51ab93be7d2a15025051db5c3 not found: ID does not exist"
Oct 30 22:16:53.174953 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.174916    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:53.174953 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.174949    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:53.174953 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.174960    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:53.175239 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.174972    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7nxb\" (UniqueName: \"kubernetes.io/projected/ebef8f6b-459e-4a01-8fa2-ed06d04208c5-kube-api-access-w7nxb\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:53.179711 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.179685    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"]
Oct 30 22:16:53.179908 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.179888    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9" podUID="b39819f5-d459-4491-a6cd-eab808ff9b36" containerName="payload-pre-processing" containerID="cri-o://6e33f8ad3c575108a4c8c619380eb4eeafb47fdc243e823ff66b30eaffa5c551" gracePeriod=30
Oct 30 22:16:53.410818 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.410766    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"]
Oct 30 22:16:53.418442 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.418413    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl"]
Oct 30 22:16:53.447353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.447324    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"
Oct 30 22:16:53.471650 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.471628    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"
Oct 30 22:16:53.580065 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.579977    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39819f5-d459-4491-a6cd-eab808ff9b36-kube-api-access-j2gnm\" (UniqueName: \"kubernetes.io/projected/b39819f5-d459-4491-a6cd-eab808ff9b36-kube-api-access-j2gnm\") pod \"b39819f5-d459-4491-a6cd-eab808ff9b36\" (UID: \"b39819f5-d459-4491-a6cd-eab808ff9b36\") "
Oct 30 22:16:53.580065 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.580058    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b277a651-334b-4560-9dc9-89bf75a1b9a6-kube-api-access-bw4b6\" (UniqueName: \"kubernetes.io/projected/b277a651-334b-4560-9dc9-89bf75a1b9a6-kube-api-access-bw4b6\") pod \"b277a651-334b-4560-9dc9-89bf75a1b9a6\" (UID: \"b277a651-334b-4560-9dc9-89bf75a1b9a6\") "
Oct 30 22:16:53.580361 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.580222    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39819f5-d459-4491-a6cd-eab808ff9b36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39819f5-d459-4491-a6cd-eab808ff9b36-plugins-config-volume\") pod \"b39819f5-d459-4491-a6cd-eab808ff9b36\" (UID: \"b39819f5-d459-4491-a6cd-eab808ff9b36\") "
Oct 30 22:16:53.580361 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.580345    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b277a651-334b-4560-9dc9-89bf75a1b9a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b277a651-334b-4560-9dc9-89bf75a1b9a6-plugins-config-volume\") pod \"b277a651-334b-4560-9dc9-89bf75a1b9a6\" (UID: \"b277a651-334b-4560-9dc9-89bf75a1b9a6\") "
Oct 30 22:16:53.580723 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.580690    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39819f5-d459-4491-a6cd-eab808ff9b36-plugins-config-volume" pod "b39819f5-d459-4491-a6cd-eab808ff9b36" (UID: "b39819f5-d459-4491-a6cd-eab808ff9b36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:53.580842 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.580727    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b277a651-334b-4560-9dc9-89bf75a1b9a6-plugins-config-volume" pod "b277a651-334b-4560-9dc9-89bf75a1b9a6" (UID: "b277a651-334b-4560-9dc9-89bf75a1b9a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:53.580904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.580888    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b277a651-334b-4560-9dc9-89bf75a1b9a6-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:53.580946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.580907    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b39819f5-d459-4491-a6cd-eab808ff9b36-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:53.583102 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.583081    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39819f5-d459-4491-a6cd-eab808ff9b36-kube-api-access-j2gnm" pod "b39819f5-d459-4491-a6cd-eab808ff9b36" (UID: "b39819f5-d459-4491-a6cd-eab808ff9b36"). InnerVolumeSpecName "kube-api-access-j2gnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:53.583252 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.583109    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b277a651-334b-4560-9dc9-89bf75a1b9a6-kube-api-access-bw4b6" pod "b277a651-334b-4560-9dc9-89bf75a1b9a6" (UID: "b277a651-334b-4560-9dc9-89bf75a1b9a6"). InnerVolumeSpecName "kube-api-access-bw4b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:53.732918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.732649    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2gnm\" (UniqueName: \"kubernetes.io/projected/b39819f5-d459-4491-a6cd-eab808ff9b36-kube-api-access-j2gnm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:53.732918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:53.732685    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw4b6\" (UniqueName: \"kubernetes.io/projected/b277a651-334b-4560-9dc9-89bf75a1b9a6-kube-api-access-bw4b6\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:54.056279 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.056250    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"
Oct 30 22:16:54.131188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.131117    4553 generic.go:356] "Generic (PLEG): container finished" podID="b39819f5-d459-4491-a6cd-eab808ff9b36" containerID="6e33f8ad3c575108a4c8c619380eb4eeafb47fdc243e823ff66b30eaffa5c551" exitCode=0
Oct 30 22:16:54.131188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.131183    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"
Oct 30 22:16:54.131455 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.131220    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9" event={"ID":"b39819f5-d459-4491-a6cd-eab808ff9b36","Type":"ContainerDied","Data":"6e33f8ad3c575108a4c8c619380eb4eeafb47fdc243e823ff66b30eaffa5c551"}
Oct 30 22:16:54.131455 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.131274    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9" event={"ID":"b39819f5-d459-4491-a6cd-eab808ff9b36","Type":"ContainerDied","Data":"85b678f82abc1280c70e8c231e76c92a4f726152d4a266385ba6e8f260e9411c"}
Oct 30 22:16:54.131455 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.131308    4553 scope.go:122] "RemoveContainer" containerID="6e33f8ad3c575108a4c8c619380eb4eeafb47fdc243e823ff66b30eaffa5c551"
Oct 30 22:16:54.132735 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.132715    4553 generic.go:356] "Generic (PLEG): container finished" podID="b277a651-334b-4560-9dc9-89bf75a1b9a6" containerID="f2ed814b5f12dec9bdbe496de1d3148f0df05a1a8438acb91a8cc313df388bad" exitCode=0
Oct 30 22:16:54.132845 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.132761    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"
Oct 30 22:16:54.132845 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.132791    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr" event={"ID":"b277a651-334b-4560-9dc9-89bf75a1b9a6","Type":"ContainerDied","Data":"f2ed814b5f12dec9bdbe496de1d3148f0df05a1a8438acb91a8cc313df388bad"}
Oct 30 22:16:54.132958 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.132881    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr" event={"ID":"b277a651-334b-4560-9dc9-89bf75a1b9a6","Type":"ContainerDied","Data":"dc688b753ebe9b51068322d65bcb910b95a8ecaeeb29f0ad97b380f09ce43779"}
Oct 30 22:16:54.143069 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.142981    4553 scope.go:122] "RemoveContainer" containerID="6e33f8ad3c575108a4c8c619380eb4eeafb47fdc243e823ff66b30eaffa5c551"
Oct 30 22:16:54.143426 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:16:54.143404    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e33f8ad3c575108a4c8c619380eb4eeafb47fdc243e823ff66b30eaffa5c551\": container with ID starting with 6e33f8ad3c575108a4c8c619380eb4eeafb47fdc243e823ff66b30eaffa5c551 not found: ID does not exist" containerID="6e33f8ad3c575108a4c8c619380eb4eeafb47fdc243e823ff66b30eaffa5c551"
Oct 30 22:16:54.143513 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.143436    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e33f8ad3c575108a4c8c619380eb4eeafb47fdc243e823ff66b30eaffa5c551"} err="failed to get container status \"6e33f8ad3c575108a4c8c619380eb4eeafb47fdc243e823ff66b30eaffa5c551\": rpc error: code = NotFound desc = could not find container \"6e33f8ad3c575108a4c8c619380eb4eeafb47fdc243e823ff66b30eaffa5c551\": container with ID starting with 6e33f8ad3c575108a4c8c619380eb4eeafb47fdc243e823ff66b30eaffa5c551 not found: ID does not exist"
Oct 30 22:16:54.143513 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.143462    4553 scope.go:122] "RemoveContainer" containerID="f2ed814b5f12dec9bdbe496de1d3148f0df05a1a8438acb91a8cc313df388bad"
Oct 30 22:16:54.152507 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.152488    4553 scope.go:122] "RemoveContainer" containerID="f2ed814b5f12dec9bdbe496de1d3148f0df05a1a8438acb91a8cc313df388bad"
Oct 30 22:16:54.152805 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:16:54.152785    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ed814b5f12dec9bdbe496de1d3148f0df05a1a8438acb91a8cc313df388bad\": container with ID starting with f2ed814b5f12dec9bdbe496de1d3148f0df05a1a8438acb91a8cc313df388bad not found: ID does not exist" containerID="f2ed814b5f12dec9bdbe496de1d3148f0df05a1a8438acb91a8cc313df388bad"
Oct 30 22:16:54.152884 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.152818    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ed814b5f12dec9bdbe496de1d3148f0df05a1a8438acb91a8cc313df388bad"} err="failed to get container status \"f2ed814b5f12dec9bdbe496de1d3148f0df05a1a8438acb91a8cc313df388bad\": rpc error: code = NotFound desc = could not find container \"f2ed814b5f12dec9bdbe496de1d3148f0df05a1a8438acb91a8cc313df388bad\": container with ID starting with f2ed814b5f12dec9bdbe496de1d3148f0df05a1a8438acb91a8cc313df388bad not found: ID does not exist"
Oct 30 22:16:54.292427 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.292395    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"]
Oct 30 22:16:54.295286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.295259    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-859078-7fc998f987-m9hx9"]
Oct 30 22:16:54.402432 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.402387    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"]
Oct 30 22:16:54.407841 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:54.407814    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-859078-795cf98998-mghsr"]
Oct 30 22:16:55.061676 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:55.061641    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"
Oct 30 22:16:55.393421 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:55.393388    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b277a651-334b-4560-9dc9-89bf75a1b9a6" path="/var/lib/kubelet/pods/b277a651-334b-4560-9dc9-89bf75a1b9a6/volumes"
Oct 30 22:16:55.393820 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:55.393806    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39819f5-d459-4491-a6cd-eab808ff9b36" path="/var/lib/kubelet/pods/b39819f5-d459-4491-a6cd-eab808ff9b36/volumes"
Oct 30 22:16:55.394229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:55.394211    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebef8f6b-459e-4a01-8fa2-ed06d04208c5" path="/var/lib/kubelet/pods/ebef8f6b-459e-4a01-8fa2-ed06d04208c5/volumes"
Oct 30 22:16:55.913484 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:55.913442    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-859078-bfd6b4d75-7ndnl" podUID="ebef8f6b-459e-4a01-8fa2-ed06d04208c5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.112:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:16:57.993321 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:57.993289    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"]
Oct 30 22:16:57.993907 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:57.993893    4553 cpu_manager.go:414] "containerMap: removing container" podUID="b39819f5-d459-4491-a6cd-eab808ff9b36" containerName="payload-pre-processing"
Oct 30 22:16:57.993971 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:57.993908    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39819f5-d459-4491-a6cd-eab808ff9b36" containerName="payload-pre-processing"
Oct 30 22:16:57.993971 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:57.993922    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ebef8f6b-459e-4a01-8fa2-ed06d04208c5" containerName="maas-api"
Oct 30 22:16:57.993971 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:57.993927    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebef8f6b-459e-4a01-8fa2-ed06d04208c5" containerName="maas-api"
Oct 30 22:16:57.993971 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:57.993962    4553 cpu_manager.go:414] "containerMap: removing container" podUID="b277a651-334b-4560-9dc9-89bf75a1b9a6" containerName="payload-processing"
Oct 30 22:16:57.993971 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:57.993967    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b277a651-334b-4560-9dc9-89bf75a1b9a6" containerName="payload-processing"
Oct 30 22:16:57.994120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:57.994055    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39819f5-d459-4491-a6cd-eab808ff9b36" containerName="payload-pre-processing"
Oct 30 22:16:57.994120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:57.994065    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebef8f6b-459e-4a01-8fa2-ed06d04208c5" containerName="maas-api"
Oct 30 22:16:57.994120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:57.994078    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="b277a651-334b-4560-9dc9-89bf75a1b9a6" containerName="payload-processing"
Oct 30 22:16:57.999305 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:57.999287    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.073292 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.073257    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"]
Oct 30 22:16:58.089117 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.089071    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-workload-certs\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.089341 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.089153    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-576nm\" (UniqueName: \"kubernetes.io/projected/45c033dd-33b5-4002-8161-d8a9402c7cd4-kube-api-access-576nm\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.089341 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.089210    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c033dd-33b5-4002-8161-d8a9402c7cd4-istiod-ca-cert\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.089341 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.089245    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-workload-socket\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.089341 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.089281    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-data\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.089341 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.089321    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-token\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.089626 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.089378    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-envoy\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.089626 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.089413    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-podinfo\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.089626 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.089475    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-credential-socket\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.104071 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.104047    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a28730-openshift-default-dockercfg-9s8r5\""
Oct 30 22:16:58.202562 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.202512    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-podinfo\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.202759 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.202604    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-credential-socket\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.202759 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.202648    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-workload-certs\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.202759 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.202699    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-576nm\" (UniqueName: \"kubernetes.io/projected/45c033dd-33b5-4002-8161-d8a9402c7cd4-kube-api-access-576nm\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.202759 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.202757    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c033dd-33b5-4002-8161-d8a9402c7cd4-istiod-ca-cert\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.202979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.202793    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-workload-socket\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.202979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.202841    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-data\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.202979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.202887    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-token\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.202979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.202955    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-envoy\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.208742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.208624    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-workload-certs\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.209364 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.209191    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-workload-socket\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.209537 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.209513    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-data\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.209537 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.209246    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-credential-socket\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.223045 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.210694    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c033dd-33b5-4002-8161-d8a9402c7cd4-istiod-ca-cert\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.223045 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.223018    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-envoy\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.225785 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.225722    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-podinfo\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.291968 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:16:58.291868    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:16:58.298873 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.298832    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-576nm\" (UniqueName: \"kubernetes.io/projected/45c033dd-33b5-4002-8161-d8a9402c7cd4-kube-api-access-576nm\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.308772 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.308739    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-token\") pod \"e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") " pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.325707 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.325410    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:58.778521 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.778489    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"]
Oct 30 22:16:58.816934 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:16:58.816903    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c033dd_33b5_4002_8161_d8a9402c7cd4.slice/crio-b075f567b01ca614ca885e83630fd478303d5ac39abe46d120be1127a18babfe WatchSource:0}: Error finding container b075f567b01ca614ca885e83630fd478303d5ac39abe46d120be1127a18babfe: Status 404 returned error can't find the container with id b075f567b01ca614ca885e83630fd478303d5ac39abe46d120be1127a18babfe
Oct 30 22:16:58.819422 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.819388    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:16:58.819515 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.819457    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:16:58.819515 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:58.819484    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:16:59.243484 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:59.243447    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2" event={"ID":"45c033dd-33b5-4002-8161-d8a9402c7cd4","Type":"ContainerStarted","Data":"011b3d302d36733cec9d1af32f2b3b52cffc145751e8349301f8ee911c544198"}
Oct 30 22:16:59.244158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:59.244113    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2" event={"ID":"45c033dd-33b5-4002-8161-d8a9402c7cd4","Type":"ContainerStarted","Data":"b075f567b01ca614ca885e83630fd478303d5ac39abe46d120be1127a18babfe"}
Oct 30 22:16:59.326646 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:59.326609    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:59.332695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:59.332665    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:16:59.364844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:16:59.364790    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2" podStartSLOduration=2.3647777039999998 podStartE2EDuration="2.364777704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:59.362705256 +0000 UTC m=+2070.097068287" watchObservedRunningTime="2026-10-30 22:16:59.364777704 +0000 UTC m=+2070.099140734"
Oct 30 22:17:00.249726 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:00.249694    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:17:00.250782 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:00.250760    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:17:11.835891 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:11.835858    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"]
Oct 30 22:17:11.836419 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:11.836115    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc" podUID="f827754d-0840-4429-9ce1-8d63d32e6871" containerName="istio-proxy" containerID="cri-o://8278e2943d86346bb1a7356aade97e2b63acc8f16ac161154fc343dd88670945" gracePeriod=30
Oct 30 22:17:13.766260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.766216    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"]
Oct 30 22:17:13.770849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.770826    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.836531 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.836496    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-96567a-openshift-default-dockercfg-c72rg\""
Oct 30 22:17:13.876117 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.876077    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-data\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.876117 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.876118    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-workload-socket\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.876355 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.876336    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzj7t\" (UniqueName: \"kubernetes.io/projected/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-kube-api-access-zzj7t\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.876403 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.876369    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-envoy\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.876448 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.876429    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-podinfo\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.876497 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.876486    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istiod-ca-cert\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.876532 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.876518    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-workload-certs\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.876623 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.876609    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-credential-socket\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.876703 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.876691    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-token\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.891407 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.891374    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"]
Oct 30 22:17:13.978289 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.978245    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-data\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.978580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.978559    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-workload-socket\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.979152 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.978683    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-data\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.979152 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.978984    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-workload-socket\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.979510 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.979487    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzj7t\" (UniqueName: \"kubernetes.io/projected/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-kube-api-access-zzj7t\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.979668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.979624    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-envoy\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.979668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.979665    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-podinfo\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.980882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.979729    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istiod-ca-cert\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.980882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.979768    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-workload-certs\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.980882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.979796    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-credential-socket\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.980882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.979889    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-token\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:13.980882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.980644    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-workload-certs\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:14.013161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.983646    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istiod-ca-cert\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:14.013161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:13.983918    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-credential-socket\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:14.013161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:14.006538    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-podinfo\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:14.021202 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:14.018332    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-envoy\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:14.071176 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:14.066748    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-token\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:14.080157 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:14.079388    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzj7t\" (UniqueName: \"kubernetes.io/projected/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-kube-api-access-zzj7t\") pod \"e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") " pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:14.111161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:14.110190    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:14.207607 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:14.207218    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"]
Oct 30 22:17:14.324689 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:14.324587    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"]
Oct 30 22:17:14.604365 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:14.604329    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"]
Oct 30 22:17:14.605569 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:17:14.605534    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0ae1bf_9305_4f6f_a581_475f9a8cc7e9.slice/crio-26674ba240ceedc039d9f76888df010a406ca54d2d40f1069af2bc22f88753c9 WatchSource:0}: Error finding container 26674ba240ceedc039d9f76888df010a406ca54d2d40f1069af2bc22f88753c9: Status 404 returned error can't find the container with id 26674ba240ceedc039d9f76888df010a406ca54d2d40f1069af2bc22f88753c9
Oct 30 22:17:14.608491 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:14.608454    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:17:14.608580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:14.608537    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:17:14.608580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:14.608571    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:17:15.376727 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:15.376689    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h" event={"ID":"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9","Type":"ContainerStarted","Data":"4706cadba9d79cc6d88d1638cf5540529cefb8eb496a68b4ef0259ea720ed08d"}
Oct 30 22:17:15.377171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:15.376747    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h" event={"ID":"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9","Type":"ContainerStarted","Data":"26674ba240ceedc039d9f76888df010a406ca54d2d40f1069af2bc22f88753c9"}
Oct 30 22:17:15.497287 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:15.497233    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h" podStartSLOduration=2.497216376 podStartE2EDuration="2.497216376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:15.495692905 +0000 UTC m=+2086.230055977" watchObservedRunningTime="2026-10-30 22:17:15.497216376 +0000 UTC m=+2086.231579407"
Oct 30 22:17:16.111242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:16.111206    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:16.124274 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:16.124246    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:16.382541 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:16.382452    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:16.383628 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:16.383607    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:17:17.114824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.114760    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:17:17.158058 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.158025    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-credential-socket\") pod \"f827754d-0840-4429-9ce1-8d63d32e6871\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") "
Oct 30 22:17:17.158058 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.158060    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-istio-envoy\") pod \"f827754d-0840-4429-9ce1-8d63d32e6871\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") "
Oct 30 22:17:17.158333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.158093    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f827754d-0840-4429-9ce1-8d63d32e6871-kube-api-access-qxr8d\" (UniqueName: \"kubernetes.io/projected/f827754d-0840-4429-9ce1-8d63d32e6871-kube-api-access-qxr8d\") pod \"f827754d-0840-4429-9ce1-8d63d32e6871\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") "
Oct 30 22:17:17.158333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.158161    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f827754d-0840-4429-9ce1-8d63d32e6871-istio-token\" (UniqueName: \"kubernetes.io/projected/f827754d-0840-4429-9ce1-8d63d32e6871-istio-token\") pod \"f827754d-0840-4429-9ce1-8d63d32e6871\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") "
Oct 30 22:17:17.158333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.158198    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-istio-data\") pod \"f827754d-0840-4429-9ce1-8d63d32e6871\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") "
Oct 30 22:17:17.158333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.158233    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-workload-certs\") pod \"f827754d-0840-4429-9ce1-8d63d32e6871\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") "
Oct 30 22:17:17.158333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.158297    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f827754d-0840-4429-9ce1-8d63d32e6871-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f827754d-0840-4429-9ce1-8d63d32e6871-istiod-ca-cert\") pod \"f827754d-0840-4429-9ce1-8d63d32e6871\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") "
Oct 30 22:17:17.158598 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.158361    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-credential-socket" pod "f827754d-0840-4429-9ce1-8d63d32e6871" (UID: "f827754d-0840-4429-9ce1-8d63d32e6871"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.158598 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.158429    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f827754d-0840-4429-9ce1-8d63d32e6871-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f827754d-0840-4429-9ce1-8d63d32e6871-istio-podinfo\") pod \"f827754d-0840-4429-9ce1-8d63d32e6871\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") "
Oct 30 22:17:17.158598 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.158473    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-workload-socket\") pod \"f827754d-0840-4429-9ce1-8d63d32e6871\" (UID: \"f827754d-0840-4429-9ce1-8d63d32e6871\") "
Oct 30 22:17:17.158598 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.158525    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-istio-data" pod "f827754d-0840-4429-9ce1-8d63d32e6871" (UID: "f827754d-0840-4429-9ce1-8d63d32e6871"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.158816 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.158616    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-workload-certs" pod "f827754d-0840-4429-9ce1-8d63d32e6871" (UID: "f827754d-0840-4429-9ce1-8d63d32e6871"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.158816 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.158707    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f827754d-0840-4429-9ce1-8d63d32e6871-istiod-ca-cert" pod "f827754d-0840-4429-9ce1-8d63d32e6871" (UID: "f827754d-0840-4429-9ce1-8d63d32e6871"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:17.158944 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.158828    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-workload-socket" pod "f827754d-0840-4429-9ce1-8d63d32e6871" (UID: "f827754d-0840-4429-9ce1-8d63d32e6871"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.159445 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.159420    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.159574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.159451    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.159574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.159467    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.159574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.159484    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f827754d-0840-4429-9ce1-8d63d32e6871-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.159574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.159499    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.160626 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.160596    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-istio-envoy" pod "f827754d-0840-4429-9ce1-8d63d32e6871" (UID: "f827754d-0840-4429-9ce1-8d63d32e6871"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:17.160702 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.160624    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f827754d-0840-4429-9ce1-8d63d32e6871-istio-token" pod "f827754d-0840-4429-9ce1-8d63d32e6871" (UID: "f827754d-0840-4429-9ce1-8d63d32e6871"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:17.160795 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.160773    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f827754d-0840-4429-9ce1-8d63d32e6871-istio-podinfo" pod "f827754d-0840-4429-9ce1-8d63d32e6871" (UID: "f827754d-0840-4429-9ce1-8d63d32e6871"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:17.160863 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.160845    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f827754d-0840-4429-9ce1-8d63d32e6871-kube-api-access-qxr8d" pod "f827754d-0840-4429-9ce1-8d63d32e6871" (UID: "f827754d-0840-4429-9ce1-8d63d32e6871"). InnerVolumeSpecName "kube-api-access-qxr8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:17.260570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.260536    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f827754d-0840-4429-9ce1-8d63d32e6871-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.260570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.260566    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxr8d\" (UniqueName: \"kubernetes.io/projected/f827754d-0840-4429-9ce1-8d63d32e6871-kube-api-access-qxr8d\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.260570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.260576    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f827754d-0840-4429-9ce1-8d63d32e6871-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.260881 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.260587    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f827754d-0840-4429-9ce1-8d63d32e6871-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:17.389642 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.389557    4553 generic.go:356] "Generic (PLEG): container finished" podID="f827754d-0840-4429-9ce1-8d63d32e6871" containerID="8278e2943d86346bb1a7356aade97e2b63acc8f16ac161154fc343dd88670945" exitCode=0
Oct 30 22:17:17.390020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.389800    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"
Oct 30 22:17:17.395337 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.395306    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc" event={"ID":"f827754d-0840-4429-9ce1-8d63d32e6871","Type":"ContainerDied","Data":"8278e2943d86346bb1a7356aade97e2b63acc8f16ac161154fc343dd88670945"}
Oct 30 22:17:17.395493 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.395355    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc" event={"ID":"f827754d-0840-4429-9ce1-8d63d32e6871","Type":"ContainerDied","Data":"f096871f2f2d6c9028fc66f069e623af7dcd2106d041d61fb2db52ee6964f2f5"}
Oct 30 22:17:17.395493 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.395377    4553 scope.go:122] "RemoveContainer" containerID="8278e2943d86346bb1a7356aade97e2b63acc8f16ac161154fc343dd88670945"
Oct 30 22:17:17.406591 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.406567    4553 scope.go:122] "RemoveContainer" containerID="619c55ba101ea597fe9d68698c30cb10e208dadf3aa152d3b398174fecef4b96"
Oct 30 22:17:17.416726 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.416708    4553 scope.go:122] "RemoveContainer" containerID="8278e2943d86346bb1a7356aade97e2b63acc8f16ac161154fc343dd88670945"
Oct 30 22:17:17.417000 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:17:17.416980    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8278e2943d86346bb1a7356aade97e2b63acc8f16ac161154fc343dd88670945\": container with ID starting with 8278e2943d86346bb1a7356aade97e2b63acc8f16ac161154fc343dd88670945 not found: ID does not exist" containerID="8278e2943d86346bb1a7356aade97e2b63acc8f16ac161154fc343dd88670945"
Oct 30 22:17:17.417057 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.417009    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8278e2943d86346bb1a7356aade97e2b63acc8f16ac161154fc343dd88670945"} err="failed to get container status \"8278e2943d86346bb1a7356aade97e2b63acc8f16ac161154fc343dd88670945\": rpc error: code = NotFound desc = could not find container \"8278e2943d86346bb1a7356aade97e2b63acc8f16ac161154fc343dd88670945\": container with ID starting with 8278e2943d86346bb1a7356aade97e2b63acc8f16ac161154fc343dd88670945 not found: ID does not exist"
Oct 30 22:17:17.417057 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.417028    4553 scope.go:122] "RemoveContainer" containerID="619c55ba101ea597fe9d68698c30cb10e208dadf3aa152d3b398174fecef4b96"
Oct 30 22:17:17.417349 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:17:17.417334    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619c55ba101ea597fe9d68698c30cb10e208dadf3aa152d3b398174fecef4b96\": container with ID starting with 619c55ba101ea597fe9d68698c30cb10e208dadf3aa152d3b398174fecef4b96 not found: ID does not exist" containerID="619c55ba101ea597fe9d68698c30cb10e208dadf3aa152d3b398174fecef4b96"
Oct 30 22:17:17.417403 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.417354    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619c55ba101ea597fe9d68698c30cb10e208dadf3aa152d3b398174fecef4b96"} err="failed to get container status \"619c55ba101ea597fe9d68698c30cb10e208dadf3aa152d3b398174fecef4b96\": rpc error: code = NotFound desc = could not find container \"619c55ba101ea597fe9d68698c30cb10e208dadf3aa152d3b398174fecef4b96\": container with ID starting with 619c55ba101ea597fe9d68698c30cb10e208dadf3aa152d3b398174fecef4b96 not found: ID does not exist"
Oct 30 22:17:17.558754 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.558715    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"]
Oct 30 22:17:17.562577 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:17.562554    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-859078-openshift-default-7df4c8bc79-b55rc"]
Oct 30 22:17:19.207743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.207705    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns" podUID="c9214ee9-5a57-41c0-b744-7468092863cd" containerName="payload-processing" containerID="cri-o://3d2c16d849d4300fecbe57f794eb0339fa587c381dfd5da9a8c015548a440b87" gracePeriod=25
Oct 30 22:17:19.325763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.325700    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn" podUID="43ca1e4a-16cc-44cf-b5a0-86e65294706e" containerName="payload-pre-processing" containerID="cri-o://2d83a7d0874f4ac8bc7d06ec471bfb0376ec25fc360d56d9e91d6145bcb89054" gracePeriod=25
Oct 30 22:17:19.403097 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.403065    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f827754d-0840-4429-9ce1-8d63d32e6871" path="/var/lib/kubelet/pods/f827754d-0840-4429-9ce1-8d63d32e6871/volumes"
Oct 30 22:17:19.407808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.407773    4553 generic.go:356] "Generic (PLEG): container finished" podID="c9214ee9-5a57-41c0-b744-7468092863cd" containerID="3d2c16d849d4300fecbe57f794eb0339fa587c381dfd5da9a8c015548a440b87" exitCode=0
Oct 30 22:17:19.408033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.407844    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns" event={"ID":"c9214ee9-5a57-41c0-b744-7468092863cd","Type":"ContainerDied","Data":"3d2c16d849d4300fecbe57f794eb0339fa587c381dfd5da9a8c015548a440b87"}
Oct 30 22:17:19.484763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.484735    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"
Oct 30 22:17:19.590021 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.589997    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9214ee9-5a57-41c0-b744-7468092863cd-kube-api-access-8tc9w\" (UniqueName: \"kubernetes.io/projected/c9214ee9-5a57-41c0-b744-7468092863cd-kube-api-access-8tc9w\") pod \"c9214ee9-5a57-41c0-b744-7468092863cd\" (UID: \"c9214ee9-5a57-41c0-b744-7468092863cd\") "
Oct 30 22:17:19.590182 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.590168    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9214ee9-5a57-41c0-b744-7468092863cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9214ee9-5a57-41c0-b744-7468092863cd-plugins-config-volume\") pod \"c9214ee9-5a57-41c0-b744-7468092863cd\" (UID: \"c9214ee9-5a57-41c0-b744-7468092863cd\") "
Oct 30 22:17:19.590573 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.590549    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9214ee9-5a57-41c0-b744-7468092863cd-plugins-config-volume" pod "c9214ee9-5a57-41c0-b744-7468092863cd" (UID: "c9214ee9-5a57-41c0-b744-7468092863cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:19.590847 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.590668    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9214ee9-5a57-41c0-b744-7468092863cd-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:19.592307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.592284    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9214ee9-5a57-41c0-b744-7468092863cd-kube-api-access-8tc9w" pod "c9214ee9-5a57-41c0-b744-7468092863cd" (UID: "c9214ee9-5a57-41c0-b744-7468092863cd"). InnerVolumeSpecName "kube-api-access-8tc9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:19.608816 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.608790    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"
Oct 30 22:17:19.691630 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.691597    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ca1e4a-16cc-44cf-b5a0-86e65294706e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ca1e4a-16cc-44cf-b5a0-86e65294706e-plugins-config-volume\") pod \"43ca1e4a-16cc-44cf-b5a0-86e65294706e\" (UID: \"43ca1e4a-16cc-44cf-b5a0-86e65294706e\") "
Oct 30 22:17:19.691855 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.691836    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ca1e4a-16cc-44cf-b5a0-86e65294706e-kube-api-access-2lpfw\" (UniqueName: \"kubernetes.io/projected/43ca1e4a-16cc-44cf-b5a0-86e65294706e-kube-api-access-2lpfw\") pod \"43ca1e4a-16cc-44cf-b5a0-86e65294706e\" (UID: \"43ca1e4a-16cc-44cf-b5a0-86e65294706e\") "
Oct 30 22:17:19.692065 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.692044    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ca1e4a-16cc-44cf-b5a0-86e65294706e-plugins-config-volume" pod "43ca1e4a-16cc-44cf-b5a0-86e65294706e" (UID: "43ca1e4a-16cc-44cf-b5a0-86e65294706e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:19.692392 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.692374    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ca1e4a-16cc-44cf-b5a0-86e65294706e-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:19.692441 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.692395    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tc9w\" (UniqueName: \"kubernetes.io/projected/c9214ee9-5a57-41c0-b744-7468092863cd-kube-api-access-8tc9w\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:19.694165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.694124    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ca1e4a-16cc-44cf-b5a0-86e65294706e-kube-api-access-2lpfw" pod "43ca1e4a-16cc-44cf-b5a0-86e65294706e" (UID: "43ca1e4a-16cc-44cf-b5a0-86e65294706e"). InnerVolumeSpecName "kube-api-access-2lpfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:19.793597 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:19.793516    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lpfw\" (UniqueName: \"kubernetes.io/projected/43ca1e4a-16cc-44cf-b5a0-86e65294706e-kube-api-access-2lpfw\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:20.416025 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:20.415995    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"
Oct 30 22:17:20.416025 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:20.416011    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns" event={"ID":"c9214ee9-5a57-41c0-b744-7468092863cd","Type":"ContainerDied","Data":"61635dc0bed0cec0d4cfb1b1c4bdda448b27aa3167cee4a8a21ec8e20b46ebde"}
Oct 30 22:17:20.416539 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:20.416073    4553 scope.go:122] "RemoveContainer" containerID="3d2c16d849d4300fecbe57f794eb0339fa587c381dfd5da9a8c015548a440b87"
Oct 30 22:17:20.417671 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:20.417647    4553 generic.go:356] "Generic (PLEG): container finished" podID="43ca1e4a-16cc-44cf-b5a0-86e65294706e" containerID="2d83a7d0874f4ac8bc7d06ec471bfb0376ec25fc360d56d9e91d6145bcb89054" exitCode=0
Oct 30 22:17:20.417778 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:20.417706    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"
Oct 30 22:17:20.417778 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:20.417713    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn" event={"ID":"43ca1e4a-16cc-44cf-b5a0-86e65294706e","Type":"ContainerDied","Data":"2d83a7d0874f4ac8bc7d06ec471bfb0376ec25fc360d56d9e91d6145bcb89054"}
Oct 30 22:17:20.417778 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:20.417757    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn" event={"ID":"43ca1e4a-16cc-44cf-b5a0-86e65294706e","Type":"ContainerDied","Data":"906b35af86eee1d203d4fac8b34893c2b122eb8cc9c034d5847b18ca572c8be5"}
Oct 30 22:17:20.426477 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:20.426455    4553 scope.go:122] "RemoveContainer" containerID="2d83a7d0874f4ac8bc7d06ec471bfb0376ec25fc360d56d9e91d6145bcb89054"
Oct 30 22:17:20.436602 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:20.436575    4553 scope.go:122] "RemoveContainer" containerID="2d83a7d0874f4ac8bc7d06ec471bfb0376ec25fc360d56d9e91d6145bcb89054"
Oct 30 22:17:20.436904 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:17:20.436884    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d83a7d0874f4ac8bc7d06ec471bfb0376ec25fc360d56d9e91d6145bcb89054\": container with ID starting with 2d83a7d0874f4ac8bc7d06ec471bfb0376ec25fc360d56d9e91d6145bcb89054 not found: ID does not exist" containerID="2d83a7d0874f4ac8bc7d06ec471bfb0376ec25fc360d56d9e91d6145bcb89054"
Oct 30 22:17:20.436953 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:20.436915    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d83a7d0874f4ac8bc7d06ec471bfb0376ec25fc360d56d9e91d6145bcb89054"} err="failed to get container status \"2d83a7d0874f4ac8bc7d06ec471bfb0376ec25fc360d56d9e91d6145bcb89054\": rpc error: code = NotFound desc = could not find container \"2d83a7d0874f4ac8bc7d06ec471bfb0376ec25fc360d56d9e91d6145bcb89054\": container with ID starting with 2d83a7d0874f4ac8bc7d06ec471bfb0376ec25fc360d56d9e91d6145bcb89054 not found: ID does not exist"
Oct 30 22:17:20.588214 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:20.588174    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"]
Oct 30 22:17:20.592388 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:20.592358    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-66cfb8678c-4vtvn"]
Oct 30 22:17:20.712672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:20.712627    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"]
Oct 30 22:17:20.717374 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:20.717344    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2edb49-67f4b9854f-zzgns"]
Oct 30 22:17:21.395603 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:21.395563    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ca1e4a-16cc-44cf-b5a0-86e65294706e" path="/var/lib/kubelet/pods/43ca1e4a-16cc-44cf-b5a0-86e65294706e/volumes"
Oct 30 22:17:21.396223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:21.396207    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9214ee9-5a57-41c0-b744-7468092863cd" path="/var/lib/kubelet/pods/c9214ee9-5a57-41c0-b744-7468092863cd/volumes"
Oct 30 22:17:23.511023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.510992    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"]
Oct 30 22:17:23.511830 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.511808    4553 cpu_manager.go:414] "containerMap: removing container" podUID="c9214ee9-5a57-41c0-b744-7468092863cd" containerName="payload-processing"
Oct 30 22:17:23.511929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.511833    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9214ee9-5a57-41c0-b744-7468092863cd" containerName="payload-processing"
Oct 30 22:17:23.511929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.511870    4553 cpu_manager.go:414] "containerMap: removing container" podUID="f827754d-0840-4429-9ce1-8d63d32e6871" containerName="istio-proxy"
Oct 30 22:17:23.511929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.511878    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f827754d-0840-4429-9ce1-8d63d32e6871" containerName="istio-proxy"
Oct 30 22:17:23.511929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.511897    4553 cpu_manager.go:414] "containerMap: removing container" podUID="f827754d-0840-4429-9ce1-8d63d32e6871" containerName="istio-proxy"
Oct 30 22:17:23.511929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.511904    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f827754d-0840-4429-9ce1-8d63d32e6871" containerName="istio-proxy"
Oct 30 22:17:23.511929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.511927    4553 cpu_manager.go:414] "containerMap: removing container" podUID="43ca1e4a-16cc-44cf-b5a0-86e65294706e" containerName="payload-pre-processing"
Oct 30 22:17:23.512260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.511935    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ca1e4a-16cc-44cf-b5a0-86e65294706e" containerName="payload-pre-processing"
Oct 30 22:17:23.512260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.512110    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="f827754d-0840-4429-9ce1-8d63d32e6871" containerName="istio-proxy"
Oct 30 22:17:23.512260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.512129    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="f827754d-0840-4429-9ce1-8d63d32e6871" containerName="istio-proxy"
Oct 30 22:17:23.512260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.512163    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ca1e4a-16cc-44cf-b5a0-86e65294706e" containerName="payload-pre-processing"
Oct 30 22:17:23.512260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.512179    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9214ee9-5a57-41c0-b744-7468092863cd" containerName="payload-processing"
Oct 30 22:17:23.515751 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.515727    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"
Oct 30 22:17:23.566642 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.566602    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcd268a-e725-4a11-9c1d-1e27977c7b22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g\" (UID: \"ffcd268a-e725-4a11-9c1d-1e27977c7b22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"
Oct 30 22:17:23.566852 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.566828    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q7ts\" (UniqueName: \"kubernetes.io/projected/ffcd268a-e725-4a11-9c1d-1e27977c7b22-kube-api-access-8q7ts\") pod \"payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g\" (UID: \"ffcd268a-e725-4a11-9c1d-1e27977c7b22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"
Oct 30 22:17:23.572292 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.572260    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"]
Oct 30 22:17:23.670424 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.670388    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-96567a\""
Oct 30 22:17:23.670934 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.670897    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-96567a-dockercfg-46bb8\""
Oct 30 22:17:23.744045 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.743661    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q7ts\" (UniqueName: \"kubernetes.io/projected/ffcd268a-e725-4a11-9c1d-1e27977c7b22-kube-api-access-8q7ts\") pod \"payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g\" (UID: \"ffcd268a-e725-4a11-9c1d-1e27977c7b22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"
Oct 30 22:17:23.744045 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.743776    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcd268a-e725-4a11-9c1d-1e27977c7b22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g\" (UID: \"ffcd268a-e725-4a11-9c1d-1e27977c7b22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"
Oct 30 22:17:23.748743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.748706    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcd268a-e725-4a11-9c1d-1e27977c7b22-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g\" (UID: \"ffcd268a-e725-4a11-9c1d-1e27977c7b22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"
Oct 30 22:17:23.763304 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.763199    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"]
Oct 30 22:17:23.779675 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.779622    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"
Oct 30 22:17:23.780501 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.780449    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"]
Oct 30 22:17:23.804394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.804361    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q7ts\" (UniqueName: \"kubernetes.io/projected/ffcd268a-e725-4a11-9c1d-1e27977c7b22-kube-api-access-8q7ts\") pod \"payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g\" (UID: \"ffcd268a-e725-4a11-9c1d-1e27977c7b22\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"
Oct 30 22:17:23.830012 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.829979    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"
Oct 30 22:17:23.844763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.844727    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bcj\" (UniqueName: \"kubernetes.io/projected/4190a170-7787-4d84-ac60-dfcfc0f33e32-kube-api-access-c8bcj\") pod \"payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk\" (UID: \"4190a170-7787-4d84-ac60-dfcfc0f33e32\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"
Oct 30 22:17:23.844903 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.844867    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4190a170-7787-4d84-ac60-dfcfc0f33e32-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk\" (UID: \"4190a170-7787-4d84-ac60-dfcfc0f33e32\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"
Oct 30 22:17:23.945877 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.945847    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4190a170-7787-4d84-ac60-dfcfc0f33e32-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk\" (UID: \"4190a170-7787-4d84-ac60-dfcfc0f33e32\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"
Oct 30 22:17:23.946021 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.945993    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bcj\" (UniqueName: \"kubernetes.io/projected/4190a170-7787-4d84-ac60-dfcfc0f33e32-kube-api-access-c8bcj\") pod \"payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk\" (UID: \"4190a170-7787-4d84-ac60-dfcfc0f33e32\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"
Oct 30 22:17:23.946555 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:23.946533    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4190a170-7787-4d84-ac60-dfcfc0f33e32-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk\" (UID: \"4190a170-7787-4d84-ac60-dfcfc0f33e32\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"
Oct 30 22:17:24.004491 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:24.004461    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bcj\" (UniqueName: \"kubernetes.io/projected/4190a170-7787-4d84-ac60-dfcfc0f33e32-kube-api-access-c8bcj\") pod \"payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk\" (UID: \"4190a170-7787-4d84-ac60-dfcfc0f33e32\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"
Oct 30 22:17:24.104656 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:24.104623    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"
Oct 30 22:17:24.212307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:24.212198    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"]
Oct 30 22:17:24.216791 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:17:24.216755    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcd268a_e725_4a11_9c1d_1e27977c7b22.slice/crio-1ed3eae21c9bbe09f925763148663ebb78a5e11e50d80388a0b1f693a216b280 WatchSource:0}: Error finding container 1ed3eae21c9bbe09f925763148663ebb78a5e11e50d80388a0b1f693a216b280: Status 404 returned error can't find the container with id 1ed3eae21c9bbe09f925763148663ebb78a5e11e50d80388a0b1f693a216b280
Oct 30 22:17:24.457156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:24.457056    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"]
Oct 30 22:17:24.457339 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:24.457303    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p" podUID="5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" containerName="istio-proxy" containerID="cri-o://155a54961fd6e67e31e1f4564fc25d8d1106fec67decba7d28c8c4c536ac7f58" gracePeriod=30
Oct 30 22:17:24.476645 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:24.476591    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g" event={"ID":"ffcd268a-e725-4a11-9c1d-1e27977c7b22","Type":"ContainerStarted","Data":"4d159233fbc4b83e81b073fcab394183dfc2ecdab3af03c91fbf07863cc2166b"}
Oct 30 22:17:24.476852 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:24.476666    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g" event={"ID":"ffcd268a-e725-4a11-9c1d-1e27977c7b22","Type":"ContainerStarted","Data":"1ed3eae21c9bbe09f925763148663ebb78a5e11e50d80388a0b1f693a216b280"}
Oct 30 22:17:24.477528 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:24.477506    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"
Oct 30 22:17:24.486945 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:24.486914    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"]
Oct 30 22:17:24.491560 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:17:24.491534    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4190a170_7787_4d84_ac60_dfcfc0f33e32.slice/crio-d40b4eb06f4a0b0dee2173b8986df0da98a160f9f4018646affe601606924d0e WatchSource:0}: Error finding container d40b4eb06f4a0b0dee2173b8986df0da98a160f9f4018646affe601606924d0e: Status 404 returned error can't find the container with id d40b4eb06f4a0b0dee2173b8986df0da98a160f9f4018646affe601606924d0e
Oct 30 22:17:24.699168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:24.699097    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g" podStartSLOduration=1.699083795 podStartE2EDuration="1.699083795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:24.696688148 +0000 UTC m=+2095.431051180" watchObservedRunningTime="2026-10-30 22:17:24.699083795 +0000 UTC m=+2095.433446823"
Oct 30 22:17:25.485043 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:25.484994    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk" event={"ID":"4190a170-7787-4d84-ac60-dfcfc0f33e32","Type":"ContainerStarted","Data":"aa9dda03ac8a945ae0bc01548de226ee69781eed3eb0238ebeff8cbc72d2a33d"}
Oct 30 22:17:25.485257 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:25.485057    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk" event={"ID":"4190a170-7787-4d84-ac60-dfcfc0f33e32","Type":"ContainerStarted","Data":"d40b4eb06f4a0b0dee2173b8986df0da98a160f9f4018646affe601606924d0e"}
Oct 30 22:17:25.485257 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:25.485097    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"
Oct 30 22:17:25.602753 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:25.602689    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk" podStartSLOduration=2.60267251 podStartE2EDuration="2.60267251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:25.60170006 +0000 UTC m=+2096.336063091" watchObservedRunningTime="2026-10-30 22:17:25.60267251 +0000 UTC m=+2096.337035555"
Oct 30 22:17:28.202809 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:17:28.202771    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:17:28.542629 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.542551    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"]
Oct 30 22:17:28.546959 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.546941    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"
Oct 30 22:17:28.602524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.602491    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"]
Oct 30 22:17:28.656090 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.655945    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d66ae6-dockercfg-gwsz5\""
Oct 30 22:17:28.732160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.662535    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c16f4fde-a1b5-41a4-b045-b71cc29d500f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2\" (UID: \"c16f4fde-a1b5-41a4-b045-b71cc29d500f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"
Oct 30 22:17:28.732160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.666034    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d66ae6\""
Oct 30 22:17:28.732160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.703397    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"]
Oct 30 22:17:28.744766 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.744736    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp6wz\" (UniqueName: \"kubernetes.io/projected/c16f4fde-a1b5-41a4-b045-b71cc29d500f-kube-api-access-hp6wz\") pod \"payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2\" (UID: \"c16f4fde-a1b5-41a4-b045-b71cc29d500f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"
Oct 30 22:17:28.750423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.750391    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"
Oct 30 22:17:28.787721 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.787669    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"]
Oct 30 22:17:28.847063 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.847030    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65182276-f498-4e0a-b778-46315447c3dd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl\" (UID: \"65182276-f498-4e0a-b778-46315447c3dd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"
Oct 30 22:17:28.847298 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.847250    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmqwx\" (UniqueName: \"kubernetes.io/projected/65182276-f498-4e0a-b778-46315447c3dd-kube-api-access-vmqwx\") pod \"payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl\" (UID: \"65182276-f498-4e0a-b778-46315447c3dd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"
Oct 30 22:17:28.847378 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.847354    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c16f4fde-a1b5-41a4-b045-b71cc29d500f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2\" (UID: \"c16f4fde-a1b5-41a4-b045-b71cc29d500f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"
Oct 30 22:17:28.847463 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.847451    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp6wz\" (UniqueName: \"kubernetes.io/projected/c16f4fde-a1b5-41a4-b045-b71cc29d500f-kube-api-access-hp6wz\") pod \"payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2\" (UID: \"c16f4fde-a1b5-41a4-b045-b71cc29d500f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"
Oct 30 22:17:28.848047 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.848021    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c16f4fde-a1b5-41a4-b045-b71cc29d500f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2\" (UID: \"c16f4fde-a1b5-41a4-b045-b71cc29d500f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"
Oct 30 22:17:28.905633 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.905598    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp6wz\" (UniqueName: \"kubernetes.io/projected/c16f4fde-a1b5-41a4-b045-b71cc29d500f-kube-api-access-hp6wz\") pod \"payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2\" (UID: \"c16f4fde-a1b5-41a4-b045-b71cc29d500f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"
Oct 30 22:17:28.949277 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.949240    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65182276-f498-4e0a-b778-46315447c3dd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl\" (UID: \"65182276-f498-4e0a-b778-46315447c3dd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"
Oct 30 22:17:28.949494 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.949367    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmqwx\" (UniqueName: \"kubernetes.io/projected/65182276-f498-4e0a-b778-46315447c3dd-kube-api-access-vmqwx\") pod \"payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl\" (UID: \"65182276-f498-4e0a-b778-46315447c3dd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"
Oct 30 22:17:28.949961 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:28.949940    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65182276-f498-4e0a-b778-46315447c3dd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl\" (UID: \"65182276-f498-4e0a-b778-46315447c3dd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"
Oct 30 22:17:29.007776 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.007744    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmqwx\" (UniqueName: \"kubernetes.io/projected/65182276-f498-4e0a-b778-46315447c3dd-kube-api-access-vmqwx\") pod \"payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl\" (UID: \"65182276-f498-4e0a-b778-46315447c3dd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"
Oct 30 22:17:29.094432 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.094395    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"
Oct 30 22:17:29.161867 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.161612    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"
Oct 30 22:17:29.461028 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.460956    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:17:29.461028 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.461016    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p" podUID="5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:17:29.472954 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:17:29.472933    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65182276_f498_4e0a_b778_46315447c3dd.slice/crio-0ec0f8d755e2439bf02e428787e23e24e1f5bbf96e4a093e91098c5a6c6ddf60 WatchSource:0}: Error finding container 0ec0f8d755e2439bf02e428787e23e24e1f5bbf96e4a093e91098c5a6c6ddf60: Status 404 returned error can't find the container with id 0ec0f8d755e2439bf02e428787e23e24e1f5bbf96e4a093e91098c5a6c6ddf60
Oct 30 22:17:29.473055 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.473036    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"]
Oct 30 22:17:29.522170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.522102    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl" event={"ID":"65182276-f498-4e0a-b778-46315447c3dd","Type":"ContainerStarted","Data":"0ec0f8d755e2439bf02e428787e23e24e1f5bbf96e4a093e91098c5a6c6ddf60"}
Oct 30 22:17:29.583352 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.583312    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"]
Oct 30 22:17:29.583730 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:17:29.583702    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16f4fde_a1b5_41a4_b045_b71cc29d500f.slice/crio-363d9a30615dd0d8f4357f0f57e331255db8bd09dd6ad66a6f46416f9df7dad5 WatchSource:0}: Error finding container 363d9a30615dd0d8f4357f0f57e331255db8bd09dd6ad66a6f46416f9df7dad5: Status 404 returned error can't find the container with id 363d9a30615dd0d8f4357f0f57e331255db8bd09dd6ad66a6f46416f9df7dad5
Oct 30 22:17:29.717014 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.716947    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:17:29.751122 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.751090    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:17:29.753534 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.753514    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:17:29.862304 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.862275    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-workload-certs\") pod \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") "
Oct 30 22:17:29.862465 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.862341    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-podinfo\") pod \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") "
Oct 30 22:17:29.862465 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.862393    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-data\") pod \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") "
Oct 30 22:17:29.862465 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.862435    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-token\" (UniqueName: \"kubernetes.io/projected/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-token\") pod \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") "
Oct 30 22:17:29.862592 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.862510    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istiod-ca-cert\") pod \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") "
Oct 30 22:17:29.862592 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.862546    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-kube-api-access-ffn2f\" (UniqueName: \"kubernetes.io/projected/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-kube-api-access-ffn2f\") pod \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") "
Oct 30 22:17:29.862684 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.862616    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-workload-certs" pod "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" (UID: "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:29.862684 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.862647    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-workload-socket\") pod \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") "
Oct 30 22:17:29.862762 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.862746    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-credential-socket\") pod \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") "
Oct 30 22:17:29.862816 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.862778    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-envoy\") pod \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\" (UID: \"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5\") "
Oct 30 22:17:29.863008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.862985    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-data" pod "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" (UID: "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:29.863183 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.863115    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istiod-ca-cert" pod "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" (UID: "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:29.863560 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.863484    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:29.863560 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.863513    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:29.863560 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.863528    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:29.863885 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.863859    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-credential-socket" pod "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" (UID: "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:29.863985 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.863886    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-workload-socket" pod "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" (UID: "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:29.865256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.865231    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-token" pod "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" (UID: "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:29.865256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.865248    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-envoy" pod "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" (UID: "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:17:29.865422 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.865349    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-podinfo" pod "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" (UID: "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:17:29.865908 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.865893    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-kube-api-access-ffn2f" pod "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" (UID: "5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5"). InnerVolumeSpecName "kube-api-access-ffn2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:29.965001 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.964958    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:29.965001 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.964992    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:29.965001 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.965006    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffn2f\" (UniqueName: \"kubernetes.io/projected/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-kube-api-access-ffn2f\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:29.965344 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.965020    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:29.965344 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.965035    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:29.965344 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:29.965048    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:30.106460 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:30.106428    4553 scope.go:122] "RemoveContainer" containerID="155a54961fd6e67e31e1f4564fc25d8d1106fec67decba7d28c8c4c536ac7f58"
Oct 30 22:17:30.530524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:30.530416    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl" event={"ID":"65182276-f498-4e0a-b778-46315447c3dd","Type":"ContainerStarted","Data":"8778fa6dad24532011d777e30a34418ba5c561507d2cf0aeb881a0186aa0e3cb"}
Oct 30 22:17:30.530524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:30.530485    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"
Oct 30 22:17:30.532086 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:30.532060    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2" event={"ID":"c16f4fde-a1b5-41a4-b045-b71cc29d500f","Type":"ContainerStarted","Data":"fc57e93bb3953182bebe9288432e22d68aa4d70e453c5ab4dfacbbe4d56a9b43"}
Oct 30 22:17:30.532253 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:30.532114    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2" event={"ID":"c16f4fde-a1b5-41a4-b045-b71cc29d500f","Type":"ContainerStarted","Data":"363d9a30615dd0d8f4357f0f57e331255db8bd09dd6ad66a6f46416f9df7dad5"}
Oct 30 22:17:30.532253 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:30.532162    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p" event={"ID":"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5","Type":"ContainerDied","Data":"155a54961fd6e67e31e1f4564fc25d8d1106fec67decba7d28c8c4c536ac7f58"}
Oct 30 22:17:30.532253 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:30.532196    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"
Oct 30 22:17:30.532253 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:30.532201    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"
Oct 30 22:17:30.532253 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:30.532217    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p" event={"ID":"5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5","Type":"ContainerDied","Data":"5199d3687cee2889c9e9b1862ff6ec7a87949043f5442b4a83bfea4fe3d98012"}
Oct 30 22:17:30.697350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:30.697294    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl" podStartSLOduration=2.697278755 podStartE2EDuration="2.697278755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:30.645432851 +0000 UTC m=+2101.379795881" watchObservedRunningTime="2026-10-30 22:17:30.697278755 +0000 UTC m=+2101.431641786"
Oct 30 22:17:30.861216 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:30.861181    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"]
Oct 30 22:17:30.864808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:30.864783    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2jl2p"]
Oct 30 22:17:30.977773 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:30.977715    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2" podStartSLOduration=2.977700633 podStartE2EDuration="2.977700633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:30.975863768 +0000 UTC m=+2101.710226799" watchObservedRunningTime="2026-10-30 22:17:30.977700633 +0000 UTC m=+2101.712063664"
Oct 30 22:17:31.393656 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:31.393611    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" path="/var/lib/kubelet/pods/5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5/volumes"
Oct 30 22:17:36.495924 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:36.495882    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"
Oct 30 22:17:36.496448 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:36.496259    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"
Oct 30 22:17:37.778160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:37.777604    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"]
Oct 30 22:17:37.896193 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:37.896098    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"]
Oct 30 22:17:41.541563 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:41.541533    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"
Oct 30 22:17:41.541990 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:41.541600    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"
Oct 30 22:17:42.777936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:42.777898    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682" podUID="3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce" containerName="payload-processing" containerID="cri-o://3054f3a576e876c09aa829c76d7c29993896c4c5b584c5de4cebeae8dfbdde40" gracePeriod=25
Oct 30 22:17:42.896668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:42.896532    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km" podUID="0345916d-40ea-4739-985d-677b545547d7" containerName="payload-pre-processing" containerID="cri-o://59eafa814c3ef6f4938d5277f88cb022f3da4cdab74908c693d255b04dc3617e" gracePeriod=25
Oct 30 22:17:43.061100 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.061076    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"
Oct 30 22:17:43.108813 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.108780    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce-kube-api-access-qbk5g\" (UniqueName: \"kubernetes.io/projected/3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce-kube-api-access-qbk5g\") pod \"3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce\" (UID: \"3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce\") "
Oct 30 22:17:43.108978 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.108893    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce-plugins-config-volume\") pod \"3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce\" (UID: \"3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce\") "
Oct 30 22:17:43.109626 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.109566    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce-plugins-config-volume" pod "3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce" (UID: "3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:43.111721 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.111668    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce-kube-api-access-qbk5g" pod "3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce" (UID: "3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce"). InnerVolumeSpecName "kube-api-access-qbk5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:43.168213 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.168185    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"
Oct 30 22:17:43.211627 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.211589    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0345916d-40ea-4739-985d-677b545547d7-kube-api-access-2w9rn\" (UniqueName: \"kubernetes.io/projected/0345916d-40ea-4739-985d-677b545547d7-kube-api-access-2w9rn\") pod \"0345916d-40ea-4739-985d-677b545547d7\" (UID: \"0345916d-40ea-4739-985d-677b545547d7\") "
Oct 30 22:17:43.211840 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.211765    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0345916d-40ea-4739-985d-677b545547d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0345916d-40ea-4739-985d-677b545547d7-plugins-config-volume\") pod \"0345916d-40ea-4739-985d-677b545547d7\" (UID: \"0345916d-40ea-4739-985d-677b545547d7\") "
Oct 30 22:17:43.212382 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.212360    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbk5g\" (UniqueName: \"kubernetes.io/projected/3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce-kube-api-access-qbk5g\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:43.212382 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.212384    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:43.214403 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.214371    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0345916d-40ea-4739-985d-677b545547d7-plugins-config-volume" pod "0345916d-40ea-4739-985d-677b545547d7" (UID: "0345916d-40ea-4739-985d-677b545547d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:43.218206 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.218170    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0345916d-40ea-4739-985d-677b545547d7-kube-api-access-2w9rn" pod "0345916d-40ea-4739-985d-677b545547d7" (UID: "0345916d-40ea-4739-985d-677b545547d7"). InnerVolumeSpecName "kube-api-access-2w9rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:43.313451 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.313375    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w9rn\" (UniqueName: \"kubernetes.io/projected/0345916d-40ea-4739-985d-677b545547d7-kube-api-access-2w9rn\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:43.313451 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.313406    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0345916d-40ea-4739-985d-677b545547d7-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:17:43.630094 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.630055    4553 generic.go:356] "Generic (PLEG): container finished" podID="0345916d-40ea-4739-985d-677b545547d7" containerID="59eafa814c3ef6f4938d5277f88cb022f3da4cdab74908c693d255b04dc3617e" exitCode=0
Oct 30 22:17:43.630428 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.630191    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"
Oct 30 22:17:43.630428 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.630196    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km" event={"ID":"0345916d-40ea-4739-985d-677b545547d7","Type":"ContainerDied","Data":"59eafa814c3ef6f4938d5277f88cb022f3da4cdab74908c693d255b04dc3617e"}
Oct 30 22:17:43.630428 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.630270    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km" event={"ID":"0345916d-40ea-4739-985d-677b545547d7","Type":"ContainerDied","Data":"162ecfc661e682e7d3c28f274e12a0a9ebd5cc62ec6b93b0b6086432e573c398"}
Oct 30 22:17:43.630428 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.630288    4553 scope.go:122] "RemoveContainer" containerID="59eafa814c3ef6f4938d5277f88cb022f3da4cdab74908c693d255b04dc3617e"
Oct 30 22:17:43.631905 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.631874    4553 generic.go:356] "Generic (PLEG): container finished" podID="3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce" containerID="3054f3a576e876c09aa829c76d7c29993896c4c5b584c5de4cebeae8dfbdde40" exitCode=0
Oct 30 22:17:43.632044 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.631958    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"
Oct 30 22:17:43.632044 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.631957    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682" event={"ID":"3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce","Type":"ContainerDied","Data":"3054f3a576e876c09aa829c76d7c29993896c4c5b584c5de4cebeae8dfbdde40"}
Oct 30 22:17:43.632044 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.632018    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682" event={"ID":"3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce","Type":"ContainerDied","Data":"e781af2b268ad8255a92c01c1aee2023fa0dd7b047c23eba1236316a9d7c248b"}
Oct 30 22:17:43.641605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.641577    4553 scope.go:122] "RemoveContainer" containerID="59eafa814c3ef6f4938d5277f88cb022f3da4cdab74908c693d255b04dc3617e"
Oct 30 22:17:43.642005 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:17:43.641980    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59eafa814c3ef6f4938d5277f88cb022f3da4cdab74908c693d255b04dc3617e\": container with ID starting with 59eafa814c3ef6f4938d5277f88cb022f3da4cdab74908c693d255b04dc3617e not found: ID does not exist" containerID="59eafa814c3ef6f4938d5277f88cb022f3da4cdab74908c693d255b04dc3617e"
Oct 30 22:17:43.642059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.642017    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59eafa814c3ef6f4938d5277f88cb022f3da4cdab74908c693d255b04dc3617e"} err="failed to get container status \"59eafa814c3ef6f4938d5277f88cb022f3da4cdab74908c693d255b04dc3617e\": rpc error: code = NotFound desc = could not find container \"59eafa814c3ef6f4938d5277f88cb022f3da4cdab74908c693d255b04dc3617e\": container with ID starting with 59eafa814c3ef6f4938d5277f88cb022f3da4cdab74908c693d255b04dc3617e not found: ID does not exist"
Oct 30 22:17:43.642059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.642037    4553 scope.go:122] "RemoveContainer" containerID="3054f3a576e876c09aa829c76d7c29993896c4c5b584c5de4cebeae8dfbdde40"
Oct 30 22:17:43.652834 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.652808    4553 scope.go:122] "RemoveContainer" containerID="3054f3a576e876c09aa829c76d7c29993896c4c5b584c5de4cebeae8dfbdde40"
Oct 30 22:17:43.653324 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:17:43.653293    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3054f3a576e876c09aa829c76d7c29993896c4c5b584c5de4cebeae8dfbdde40\": container with ID starting with 3054f3a576e876c09aa829c76d7c29993896c4c5b584c5de4cebeae8dfbdde40 not found: ID does not exist" containerID="3054f3a576e876c09aa829c76d7c29993896c4c5b584c5de4cebeae8dfbdde40"
Oct 30 22:17:43.653462 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.653335    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3054f3a576e876c09aa829c76d7c29993896c4c5b584c5de4cebeae8dfbdde40"} err="failed to get container status \"3054f3a576e876c09aa829c76d7c29993896c4c5b584c5de4cebeae8dfbdde40\": rpc error: code = NotFound desc = could not find container \"3054f3a576e876c09aa829c76d7c29993896c4c5b584c5de4cebeae8dfbdde40\": container with ID starting with 3054f3a576e876c09aa829c76d7c29993896c4c5b584c5de4cebeae8dfbdde40 not found: ID does not exist"
Oct 30 22:17:43.800499 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.800460    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"]
Oct 30 22:17:43.850715 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.850677    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-8fb7c9795-p84km"]
Oct 30 22:17:43.931607 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.931519    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"]
Oct 30 22:17:43.941618 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:43.941583    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-577ea8-7987bc78f7-p6682"]
Oct 30 22:17:45.393535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:45.393501    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0345916d-40ea-4739-985d-677b545547d7" path="/var/lib/kubelet/pods/0345916d-40ea-4739-985d-677b545547d7/volumes"
Oct 30 22:17:45.393963 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:45.393943    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce" path="/var/lib/kubelet/pods/3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce/volumes"
Oct 30 22:17:58.202507 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:17:58.202469    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:17:59.991546 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:59.991510    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"]
Oct 30 22:17:59.992116 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:59.992103    4553 cpu_manager.go:414] "containerMap: removing container" podUID="3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce" containerName="payload-processing"
Oct 30 22:17:59.992187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:59.992117    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce" containerName="payload-processing"
Oct 30 22:17:59.992187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:59.992165    4553 cpu_manager.go:414] "containerMap: removing container" podUID="5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" containerName="istio-proxy"
Oct 30 22:17:59.992187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:59.992171    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" containerName="istio-proxy"
Oct 30 22:17:59.992187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:59.992179    4553 cpu_manager.go:414] "containerMap: removing container" podUID="0345916d-40ea-4739-985d-677b545547d7" containerName="payload-pre-processing"
Oct 30 22:17:59.992187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:59.992184    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0345916d-40ea-4739-985d-677b545547d7" containerName="payload-pre-processing"
Oct 30 22:17:59.992343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:59.992280    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d10f5a1-0ada-4aa7-b9d0-b0fc2ad958ce" containerName="payload-processing"
Oct 30 22:17:59.992343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:59.992294    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9bfbd5-1f38-4154-b3a3-d7ad054fe4e5" containerName="istio-proxy"
Oct 30 22:17:59.992343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:59.992300    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="0345916d-40ea-4739-985d-677b545547d7" containerName="payload-pre-processing"
Oct 30 22:17:59.995947 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:17:59.995931    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:00.034198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.034155    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-metrics-certs\") pod \"maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:00.034383 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.034335    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:00.034383 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.034375    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621f7d89-0683-4902-8d4d-078aa96761f4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:00.034488 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.034474    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82h7c\" (UniqueName: \"kubernetes.io/projected/621f7d89-0683-4902-8d4d-078aa96761f4-kube-api-access-82h7c\") pod \"maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:00.099910 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.099876    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a28730-serving-cert\""
Oct 30 22:18:00.103214 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.103185    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"]
Oct 30 22:18:00.135278 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.135239    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:00.135465 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.135289    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621f7d89-0683-4902-8d4d-078aa96761f4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:00.135465 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.135334    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82h7c\" (UniqueName: \"kubernetes.io/projected/621f7d89-0683-4902-8d4d-078aa96761f4-kube-api-access-82h7c\") pod \"maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:00.135465 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.135399    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-metrics-certs\") pod \"maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:00.136018 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:18:00.135997    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-serving-cert: secret "maas-api-e2e-ipp-b-a28730-serving-cert" not found
Oct 30 22:18:00.136484 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:18:00.136459    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-maas-api-tls podName:621f7d89-0683-4902-8d4d-078aa96761f4 nodeName:}" failed. No retries permitted until 2026-10-30 22:18:00.636442305 +0000 UTC m=+2131.370805314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-maas-api-tls") pod "maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7" (UID: "621f7d89-0683-4902-8d4d-078aa96761f4") : secret "maas-api-e2e-ipp-b-a28730-serving-cert" not found
Oct 30 22:18:00.136619 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.136352    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621f7d89-0683-4902-8d4d-078aa96761f4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:00.138557 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.138530    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-metrics-certs\") pod \"maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:00.194856 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.194805    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82h7c\" (UniqueName: \"kubernetes.io/projected/621f7d89-0683-4902-8d4d-078aa96761f4-kube-api-access-82h7c\") pod \"maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:00.648375 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.648336    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:00.650948 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.650927    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:00.910455 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:00.910341    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:01.292406 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:01.291304    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"]
Oct 30 22:18:01.296892 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:01.296867    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:18:01.747444 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:01.747409    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7" event={"ID":"621f7d89-0683-4902-8d4d-078aa96761f4","Type":"ContainerStarted","Data":"2b183b059a6dcfbb27ce2b882ba82c3f00e46b8d8eed0c41313b2f86fc872a25"}
Oct 30 22:18:02.755525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:02.755483    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7" event={"ID":"621f7d89-0683-4902-8d4d-078aa96761f4","Type":"ContainerStarted","Data":"fa87c17796bf4e6acacf386f0131e3a719fb0dc2ed847e578479afe65dc92480"}
Oct 30 22:18:02.756020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:02.755578    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:02.876548 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:02.876495    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7" podStartSLOduration=3.017286108 podStartE2EDuration="3.876482977s" totalImagesPullingTime="859.196869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:02.876215169 +0000 UTC m=+2133.610578204" watchObservedRunningTime="2026-10-30 22:18:02.876482977 +0000 UTC m=+2133.610846041"
Oct 30 22:18:08.768368 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:08.768334    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:18:10.740197 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:10.740160    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"]
Oct 30 22:18:10.744605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:10.744580    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"
Oct 30 22:18:10.802544 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:10.802510    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"]
Oct 30 22:18:10.805274 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:10.805224    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwqf\" (UniqueName: \"kubernetes.io/projected/0e8c351f-703e-45db-a083-15d13b3399e4-kube-api-access-frwqf\") pod \"payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs\" (UID: \"0e8c351f-703e-45db-a083-15d13b3399e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"
Oct 30 22:18:10.805446 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:10.805430    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8c351f-703e-45db-a083-15d13b3399e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs\" (UID: \"0e8c351f-703e-45db-a083-15d13b3399e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"
Oct 30 22:18:10.848085 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:10.848051    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2edb49\""
Oct 30 22:18:10.848392 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:10.848267    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2edb49-dockercfg-8wptj\""
Oct 30 22:18:10.906573 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:10.906536    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwqf\" (UniqueName: \"kubernetes.io/projected/0e8c351f-703e-45db-a083-15d13b3399e4-kube-api-access-frwqf\") pod \"payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs\" (UID: \"0e8c351f-703e-45db-a083-15d13b3399e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"
Oct 30 22:18:10.906794 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:10.906695    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8c351f-703e-45db-a083-15d13b3399e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs\" (UID: \"0e8c351f-703e-45db-a083-15d13b3399e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"
Oct 30 22:18:10.907581 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:10.907560    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8c351f-703e-45db-a083-15d13b3399e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs\" (UID: \"0e8c351f-703e-45db-a083-15d13b3399e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"
Oct 30 22:18:10.965229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:10.965194    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwqf\" (UniqueName: \"kubernetes.io/projected/0e8c351f-703e-45db-a083-15d13b3399e4-kube-api-access-frwqf\") pod \"payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs\" (UID: \"0e8c351f-703e-45db-a083-15d13b3399e4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"
Oct 30 22:18:11.058503 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.058412    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"
Oct 30 22:18:11.440619 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.440588    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"]
Oct 30 22:18:11.442387 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:18:11.442360    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8c351f_703e_45db_a083_15d13b3399e4.slice/crio-1665640df7d4b4f9f55b3004b02959e880b351e9994be1701663d5da29efa8f5 WatchSource:0}: Error finding container 1665640df7d4b4f9f55b3004b02959e880b351e9994be1701663d5da29efa8f5: Status 404 returned error can't find the container with id 1665640df7d4b4f9f55b3004b02959e880b351e9994be1701663d5da29efa8f5
Oct 30 22:18:11.505606 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.505566    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"]
Oct 30 22:18:11.511116 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.511095    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"
Oct 30 22:18:11.566921 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.566885    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"]
Oct 30 22:18:11.616892 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.616858    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqvj8\" (UniqueName: \"kubernetes.io/projected/94e79eee-aaaf-49d0-b354-748bdfbe636e-kube-api-access-hqvj8\") pod \"payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj\" (UID: \"94e79eee-aaaf-49d0-b354-748bdfbe636e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"
Oct 30 22:18:11.617099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.616947    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e79eee-aaaf-49d0-b354-748bdfbe636e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj\" (UID: \"94e79eee-aaaf-49d0-b354-748bdfbe636e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"
Oct 30 22:18:11.718097 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.718010    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqvj8\" (UniqueName: \"kubernetes.io/projected/94e79eee-aaaf-49d0-b354-748bdfbe636e-kube-api-access-hqvj8\") pod \"payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj\" (UID: \"94e79eee-aaaf-49d0-b354-748bdfbe636e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"
Oct 30 22:18:11.718296 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.718099    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e79eee-aaaf-49d0-b354-748bdfbe636e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj\" (UID: \"94e79eee-aaaf-49d0-b354-748bdfbe636e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"
Oct 30 22:18:11.718894 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.718873    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e79eee-aaaf-49d0-b354-748bdfbe636e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj\" (UID: \"94e79eee-aaaf-49d0-b354-748bdfbe636e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"
Oct 30 22:18:11.776709 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.776675    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqvj8\" (UniqueName: \"kubernetes.io/projected/94e79eee-aaaf-49d0-b354-748bdfbe636e-kube-api-access-hqvj8\") pod \"payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj\" (UID: \"94e79eee-aaaf-49d0-b354-748bdfbe636e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"
Oct 30 22:18:11.822900 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.822861    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs" event={"ID":"0e8c351f-703e-45db-a083-15d13b3399e4","Type":"ContainerStarted","Data":"8d3cb77cf7c5b86d4a21da0757b84593b474c53d5aec5af608ed2567e1b2b373"}
Oct 30 22:18:11.823110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.822925    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs" event={"ID":"0e8c351f-703e-45db-a083-15d13b3399e4","Type":"ContainerStarted","Data":"1665640df7d4b4f9f55b3004b02959e880b351e9994be1701663d5da29efa8f5"}
Oct 30 22:18:11.823110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.823075    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"
Oct 30 22:18:11.825328 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.825306    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"
Oct 30 22:18:11.943776 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:11.943693    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs" podStartSLOduration=1.94367187 podStartE2EDuration="1.94367187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:11.941939718 +0000 UTC m=+2142.676302750" watchObservedRunningTime="2026-10-30 22:18:11.94367187 +0000 UTC m=+2142.678034902"
Oct 30 22:18:12.221584 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:12.221553    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"]
Oct 30 22:18:12.222628 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:18:12.222603    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e79eee_aaaf_49d0_b354_748bdfbe636e.slice/crio-05951a4115c0e046482048e0fdb8d8c4c1833ae108d3e5fa367f3c0c5bbfa874 WatchSource:0}: Error finding container 05951a4115c0e046482048e0fdb8d8c4c1833ae108d3e5fa367f3c0c5bbfa874: Status 404 returned error can't find the container with id 05951a4115c0e046482048e0fdb8d8c4c1833ae108d3e5fa367f3c0c5bbfa874
Oct 30 22:18:12.832554 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:12.832517    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj" event={"ID":"94e79eee-aaaf-49d0-b354-748bdfbe636e","Type":"ContainerStarted","Data":"0ff162e07497aa213aa412f3ef811ea529323b8ff4eb850ef4f83512d9ecba8e"}
Oct 30 22:18:12.833016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:12.832579    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj" event={"ID":"94e79eee-aaaf-49d0-b354-748bdfbe636e","Type":"ContainerStarted","Data":"05951a4115c0e046482048e0fdb8d8c4c1833ae108d3e5fa367f3c0c5bbfa874"}
Oct 30 22:18:12.833016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:12.832683    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"
Oct 30 22:18:12.952418 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:12.952367    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj" podStartSLOduration=1.9523514180000001 podStartE2EDuration="1.952351418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:12.94765698 +0000 UTC m=+2143.682020012" watchObservedRunningTime="2026-10-30 22:18:12.952351418 +0000 UTC m=+2143.686714456"
Oct 30 22:18:22.835818 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:22.835768    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"
Oct 30 22:18:23.842211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:23.842176    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"
Oct 30 22:18:27.932034 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:27.931994    4553 generic.go:356] "Generic (PLEG): container finished" podID="45c033dd-33b5-4002-8161-d8a9402c7cd4" containerID="011b3d302d36733cec9d1af32f2b3b52cffc145751e8349301f8ee911c544198" exitCode=0
Oct 30 22:18:27.932525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:27.932065    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2" event={"ID":"45c033dd-33b5-4002-8161-d8a9402c7cd4","Type":"ContainerDied","Data":"011b3d302d36733cec9d1af32f2b3b52cffc145751e8349301f8ee911c544198"}
Oct 30 22:18:27.932525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:27.932523    4553 scope.go:122] "RemoveContainer" containerID="011b3d302d36733cec9d1af32f2b3b52cffc145751e8349301f8ee911c544198"
Oct 30 22:18:27.934121 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:27.934089    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:18:27.934254 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:27.934189    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:18:27.934254 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:27.934226    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:18:28.208501 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:18:28.208391    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:18:28.674692 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:28.674657    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"]
Oct 30 22:18:28.678481 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:28.678462    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:28.733995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:28.733951    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d66ae6-serving-cert\""
Oct 30 22:18:28.738318 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:28.738279    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"]
Oct 30 22:18:28.831166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:28.830273    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-metrics-certs\") pod \"maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:28.831166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:28.830340    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27966b9b-3178-4164-8446-1115ad22399c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:28.831166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:28.830445    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk9vv\" (UniqueName: \"kubernetes.io/projected/27966b9b-3178-4164-8446-1115ad22399c-kube-api-access-qk9vv\") pod \"maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:28.831166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:28.830525    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:28.954215 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:28.931526    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:28.955000 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:28.954969    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-metrics-certs\") pod \"maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:28.955190 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:28.955173    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27966b9b-3178-4164-8446-1115ad22399c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:28.955432 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:28.955416    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk9vv\" (UniqueName: \"kubernetes.io/projected/27966b9b-3178-4164-8446-1115ad22399c-kube-api-access-qk9vv\") pod \"maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:28.955937 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:18:28.932035    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-serving-cert: secret "maas-api-e2e-praxis-a-d66ae6-serving-cert" not found
Oct 30 22:18:28.956321 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:18:28.956303    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-maas-api-tls podName:27966b9b-3178-4164-8446-1115ad22399c nodeName:}" failed. No retries permitted until 2026-10-30 22:18:29.456255384 +0000 UTC m=+2160.190618403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-maas-api-tls") pod "maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f" (UID: "27966b9b-3178-4164-8446-1115ad22399c") : secret "maas-api-e2e-praxis-a-d66ae6-serving-cert" not found
Oct 30 22:18:28.984249 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:28.959094    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27966b9b-3178-4164-8446-1115ad22399c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:28.993651 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:28.993618    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-metrics-certs\") pod \"maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:29.003644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:29.003594    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2" event={"ID":"45c033dd-33b5-4002-8161-d8a9402c7cd4","Type":"ContainerStarted","Data":"17dad28906f86142c6d81d85e3b379df72eb47d2288303f28b55b40bd0ef9d92"}
Oct 30 22:18:29.044309 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:29.044275    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk9vv\" (UniqueName: \"kubernetes.io/projected/27966b9b-3178-4164-8446-1115ad22399c-kube-api-access-qk9vv\") pod \"maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:29.328503 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:29.328343    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:18:29.330209 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:29.330178    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 22:18:29.330463 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:29.330430    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2" podUID="45c033dd-33b5-4002-8161-d8a9402c7cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 22:18:29.464283 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:29.464255    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:29.466802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:29.466777    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:29.592632 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:29.592593    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:30.002175 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:30.002127    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"]
Oct 30 22:18:30.039218 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:18:30.004419    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27966b9b_3178_4164_8446_1115ad22399c.slice/crio-9f85911c216021810999f5e503caf91ac85d92b948fcd15c949a71141d66be11 WatchSource:0}: Error finding container 9f85911c216021810999f5e503caf91ac85d92b948fcd15c949a71141d66be11: Status 404 returned error can't find the container with id 9f85911c216021810999f5e503caf91ac85d92b948fcd15c949a71141d66be11
Oct 30 22:18:30.043773 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:30.043735    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f" event={"ID":"27966b9b-3178-4164-8446-1115ad22399c","Type":"ContainerStarted","Data":"9f85911c216021810999f5e503caf91ac85d92b948fcd15c949a71141d66be11"}
Oct 30 22:18:30.249974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:30.249882    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:18:30.333291 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:30.333246    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 22:18:30.333506 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:30.333308    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2" podUID="45c033dd-33b5-4002-8161-d8a9402c7cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 22:18:31.051589 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:31.051554    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f" event={"ID":"27966b9b-3178-4164-8446-1115ad22399c","Type":"ContainerStarted","Data":"f332381de5f06906405868ddb81f81a1be846b6dee74ee058a06492b1f85b915"}
Oct 30 22:18:31.052016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:31.051655    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:31.233924 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:31.233796    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f" podStartSLOduration=2.370603127 podStartE2EDuration="3.233770002s" totalImagesPullingTime="863.166875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:31.208388773 +0000 UTC m=+2161.942751803" watchObservedRunningTime="2026-10-30 22:18:31.233770002 +0000 UTC m=+2161.968133040"
Oct 30 22:18:31.326677 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:31.326633    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 22:18:31.327023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:31.326990    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2" podUID="45c033dd-33b5-4002-8161-d8a9402c7cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 22:18:32.330569 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:32.330530    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 22:18:32.331244 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:32.331200    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2" podUID="45c033dd-33b5-4002-8161-d8a9402c7cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 22:18:33.326652 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:33.326613    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 22:18:33.326972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:33.326943    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2" podUID="45c033dd-33b5-4002-8161-d8a9402c7cd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 22:18:34.331288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:34.331252    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:18:34.332526 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:34.332492    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:18:37.066152 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:37.066108    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:18:44.457333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:44.457295    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"]
Oct 30 22:18:44.463084 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:44.463061    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"
Oct 30 22:18:44.523559 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:44.523525    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-577ea8\""
Oct 30 22:18:44.523996 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:44.523844    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"]
Oct 30 22:18:44.569466 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:44.569436    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-577ea8-dockercfg-tnq6n\""
Oct 30 22:18:44.628242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:44.628201    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxnd8\" (UniqueName: \"kubernetes.io/projected/06b02b63-62f6-4575-be94-b976bad43919-kube-api-access-sxnd8\") pod \"payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b\" (UID: \"06b02b63-62f6-4575-be94-b976bad43919\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"
Oct 30 22:18:44.628432 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:44.628396    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b02b63-62f6-4575-be94-b976bad43919-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b\" (UID: \"06b02b63-62f6-4575-be94-b976bad43919\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"
Oct 30 22:18:44.734111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:44.734022    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxnd8\" (UniqueName: \"kubernetes.io/projected/06b02b63-62f6-4575-be94-b976bad43919-kube-api-access-sxnd8\") pod \"payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b\" (UID: \"06b02b63-62f6-4575-be94-b976bad43919\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"
Oct 30 22:18:44.734543 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:44.734517    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b02b63-62f6-4575-be94-b976bad43919-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b\" (UID: \"06b02b63-62f6-4575-be94-b976bad43919\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"
Oct 30 22:18:44.735739 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:44.735712    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b02b63-62f6-4575-be94-b976bad43919-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b\" (UID: \"06b02b63-62f6-4575-be94-b976bad43919\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"
Oct 30 22:18:44.832678 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:44.832641    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxnd8\" (UniqueName: \"kubernetes.io/projected/06b02b63-62f6-4575-be94-b976bad43919-kube-api-access-sxnd8\") pod \"payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b\" (UID: \"06b02b63-62f6-4575-be94-b976bad43919\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"
Oct 30 22:18:45.089182 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:45.089153    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"
Oct 30 22:18:45.231189 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:45.231155    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"]
Oct 30 22:18:45.235681 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:45.235662    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"
Oct 30 22:18:45.296232 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:45.296201    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"]
Oct 30 22:18:45.354332 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:45.344230    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/756edb22-9e36-48e8-9835-a6570315b9cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk\" (UID: \"756edb22-9e36-48e8-9835-a6570315b9cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"
Oct 30 22:18:45.354332 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:45.344604    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hszfl\" (UniqueName: \"kubernetes.io/projected/756edb22-9e36-48e8-9835-a6570315b9cc-kube-api-access-hszfl\") pod \"payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk\" (UID: \"756edb22-9e36-48e8-9835-a6570315b9cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"
Oct 30 22:18:45.449069 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:45.449022    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/756edb22-9e36-48e8-9835-a6570315b9cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk\" (UID: \"756edb22-9e36-48e8-9835-a6570315b9cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"
Oct 30 22:18:45.449327 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:45.449126    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hszfl\" (UniqueName: \"kubernetes.io/projected/756edb22-9e36-48e8-9835-a6570315b9cc-kube-api-access-hszfl\") pod \"payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk\" (UID: \"756edb22-9e36-48e8-9835-a6570315b9cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"
Oct 30 22:18:45.450977 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:45.450940    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/756edb22-9e36-48e8-9835-a6570315b9cc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk\" (UID: \"756edb22-9e36-48e8-9835-a6570315b9cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"
Oct 30 22:18:45.474964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:45.474933    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"]
Oct 30 22:18:45.476060 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:18:45.476027    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b02b63_62f6_4575_be94_b976bad43919.slice/crio-46a5e04fd7f999f34cb197aab2cc823b048630f04422f64904489c09a88ea536 WatchSource:0}: Error finding container 46a5e04fd7f999f34cb197aab2cc823b048630f04422f64904489c09a88ea536: Status 404 returned error can't find the container with id 46a5e04fd7f999f34cb197aab2cc823b048630f04422f64904489c09a88ea536
Oct 30 22:18:45.557176 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:45.549152    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hszfl\" (UniqueName: \"kubernetes.io/projected/756edb22-9e36-48e8-9835-a6570315b9cc-kube-api-access-hszfl\") pod \"payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk\" (UID: \"756edb22-9e36-48e8-9835-a6570315b9cc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"
Oct 30 22:18:45.557176 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:45.551986    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"
Oct 30 22:18:46.010831 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:46.010796    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"]
Oct 30 22:18:46.012389 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:18:46.012357    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756edb22_9e36_48e8_9835_a6570315b9cc.slice/crio-567e15a1ed619bb133fa10f5367cc55ea530aa321afffa776fb8b2ec4895bbb1 WatchSource:0}: Error finding container 567e15a1ed619bb133fa10f5367cc55ea530aa321afffa776fb8b2ec4895bbb1: Status 404 returned error can't find the container with id 567e15a1ed619bb133fa10f5367cc55ea530aa321afffa776fb8b2ec4895bbb1
Oct 30 22:18:46.192083 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:46.192044    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk" event={"ID":"756edb22-9e36-48e8-9835-a6570315b9cc","Type":"ContainerStarted","Data":"e36763e1d4925b98d7926c5882c84ab334234c53291b6711c41946959dd58dcf"}
Oct 30 22:18:46.192321 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:46.192112    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk" event={"ID":"756edb22-9e36-48e8-9835-a6570315b9cc","Type":"ContainerStarted","Data":"567e15a1ed619bb133fa10f5367cc55ea530aa321afffa776fb8b2ec4895bbb1"}
Oct 30 22:18:46.192393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:46.192337    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"
Oct 30 22:18:46.193774 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:46.193748    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b" event={"ID":"06b02b63-62f6-4575-be94-b976bad43919","Type":"ContainerStarted","Data":"cea4f74e365209b8294d66cb98663efb4f625c976205a44965d8bf3c5930c954"}
Oct 30 22:18:46.193918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:46.193787    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b" event={"ID":"06b02b63-62f6-4575-be94-b976bad43919","Type":"ContainerStarted","Data":"46a5e04fd7f999f34cb197aab2cc823b048630f04422f64904489c09a88ea536"}
Oct 30 22:18:46.193918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:46.193847    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"
Oct 30 22:18:46.338159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:46.331620    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk" podStartSLOduration=1.331599622 podStartE2EDuration="1.331599622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:46.328492877 +0000 UTC m=+2177.062855932" watchObservedRunningTime="2026-10-30 22:18:46.331599622 +0000 UTC m=+2177.065962655"
Oct 30 22:18:46.464160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:46.454866    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b" podStartSLOduration=2.454844443 podStartE2EDuration="2.454844443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:46.453711815 +0000 UTC m=+2177.188074846" watchObservedRunningTime="2026-10-30 22:18:46.454844443 +0000 UTC m=+2177.189207475"
Oct 30 22:18:56.324702 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:56.324669    4553 generic.go:356] "Generic (PLEG): container finished" podID="8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" containerID="4706cadba9d79cc6d88d1638cf5540529cefb8eb496a68b4ef0259ea720ed08d" exitCode=0
Oct 30 22:18:56.325274 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:56.324746    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h" event={"ID":"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9","Type":"ContainerDied","Data":"4706cadba9d79cc6d88d1638cf5540529cefb8eb496a68b4ef0259ea720ed08d"}
Oct 30 22:18:56.325274 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:56.325226    4553 scope.go:122] "RemoveContainer" containerID="4706cadba9d79cc6d88d1638cf5540529cefb8eb496a68b4ef0259ea720ed08d"
Oct 30 22:18:56.326875 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:56.326843    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:18:56.327005 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:56.326933    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:18:56.327005 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:56.326982    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:18:57.203345 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:57.203313    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"
Oct 30 22:18:57.203679 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:57.203658    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"
Oct 30 22:18:57.344023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:57.343986    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h" event={"ID":"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9","Type":"ContainerStarted","Data":"7f5f959d1db7353803057a5cd6ea52c34476ed12451c8dbe67523e16dc70f42b"}
Oct 30 22:18:58.126161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:58.115526    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:18:58.133320 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:58.133284    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:18:58.133585 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:58.133495    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h" podUID="8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 22:18:58.202783 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:18:58.202748    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:18:59.127161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:59.124556    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:18:59.127161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:18:59.124623    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h" podUID="8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 22:19:00.113161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:00.112559    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:19:00.113161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:00.112639    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h" podUID="8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 22:19:01.113169 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:01.112576    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:19:01.113169 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:01.112644    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h" podUID="8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 22:19:01.383199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:01.383104    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:19:02.053579 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:02.053543    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"]
Oct 30 22:19:02.053811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:02.053788    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerName="istio-proxy" containerID="cri-o://2befb8b0a81b279f2fb31c1bc180faa0e30de8d22cd5d4258134cf6e3119d9dc" gracePeriod=30
Oct 30 22:19:02.128863 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:02.128822    4553 patch_prober.go:29] interesting pod/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:19:02.129561 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:02.129517    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h" podUID="8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 22:19:03.116429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:03.116392    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:19:03.117670 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:03.117643    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:19:07.344359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.344333    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:19:07.408811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.408772    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-token\" (UniqueName: \"kubernetes.io/projected/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-token\") pod \"a57febb1-62d4-475d-8a12-61cad4e3b93a\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") "
Oct 30 22:19:07.409069 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.408883    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-podinfo\") pod \"a57febb1-62d4-475d-8a12-61cad4e3b93a\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") "
Oct 30 22:19:07.409069 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.408952    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-credential-socket\") pod \"a57febb1-62d4-475d-8a12-61cad4e3b93a\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") "
Oct 30 22:19:07.409069 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.408990    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-workload-socket\") pod \"a57febb1-62d4-475d-8a12-61cad4e3b93a\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") "
Oct 30 22:19:07.409279 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.409104    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a57febb1-62d4-475d-8a12-61cad4e3b93a-kube-api-access-lbxft\" (UniqueName: \"kubernetes.io/projected/a57febb1-62d4-475d-8a12-61cad4e3b93a-kube-api-access-lbxft\") pod \"a57febb1-62d4-475d-8a12-61cad4e3b93a\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") "
Oct 30 22:19:07.409279 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.409250    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-data\") pod \"a57febb1-62d4-475d-8a12-61cad4e3b93a\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") "
Oct 30 22:19:07.409389 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.409323    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a57febb1-62d4-475d-8a12-61cad4e3b93a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a57febb1-62d4-475d-8a12-61cad4e3b93a-istiod-ca-cert\") pod \"a57febb1-62d4-475d-8a12-61cad4e3b93a\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") "
Oct 30 22:19:07.409389 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.409374    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-envoy\") pod \"a57febb1-62d4-475d-8a12-61cad4e3b93a\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") "
Oct 30 22:19:07.409544 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.409511    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-credential-socket" pod "a57febb1-62d4-475d-8a12-61cad4e3b93a" (UID: "a57febb1-62d4-475d-8a12-61cad4e3b93a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:07.409605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.409578    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-workload-socket" pod "a57febb1-62d4-475d-8a12-61cad4e3b93a" (UID: "a57febb1-62d4-475d-8a12-61cad4e3b93a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:07.409683 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.409660    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-workload-certs\") pod \"a57febb1-62d4-475d-8a12-61cad4e3b93a\" (UID: \"a57febb1-62d4-475d-8a12-61cad4e3b93a\") "
Oct 30 22:19:07.410237 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.409856    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-data" pod "a57febb1-62d4-475d-8a12-61cad4e3b93a" (UID: "a57febb1-62d4-475d-8a12-61cad4e3b93a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:07.410454 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.410165    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-workload-certs" pod "a57febb1-62d4-475d-8a12-61cad4e3b93a" (UID: "a57febb1-62d4-475d-8a12-61cad4e3b93a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:07.411023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.410980    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a57febb1-62d4-475d-8a12-61cad4e3b93a-istiod-ca-cert" pod "a57febb1-62d4-475d-8a12-61cad4e3b93a" (UID: "a57febb1-62d4-475d-8a12-61cad4e3b93a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:07.413442 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.413418    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:07.413602 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.413446    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:07.413602 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.413462    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:07.413602 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.413476    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a57febb1-62d4-475d-8a12-61cad4e3b93a-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:07.413602 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.413489    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:07.414391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.413656    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-envoy" pod "a57febb1-62d4-475d-8a12-61cad4e3b93a" (UID: "a57febb1-62d4-475d-8a12-61cad4e3b93a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:07.414391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.413681    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57febb1-62d4-475d-8a12-61cad4e3b93a-kube-api-access-lbxft" pod "a57febb1-62d4-475d-8a12-61cad4e3b93a" (UID: "a57febb1-62d4-475d-8a12-61cad4e3b93a"). InnerVolumeSpecName "kube-api-access-lbxft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:07.414508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.414450    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-podinfo" pod "a57febb1-62d4-475d-8a12-61cad4e3b93a" (UID: "a57febb1-62d4-475d-8a12-61cad4e3b93a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:19:07.415166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.415106    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-token" pod "a57febb1-62d4-475d-8a12-61cad4e3b93a" (UID: "a57febb1-62d4-475d-8a12-61cad4e3b93a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:07.420764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.420738    4553 generic.go:356] "Generic (PLEG): container finished" podID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerID="2befb8b0a81b279f2fb31c1bc180faa0e30de8d22cd5d4258134cf6e3119d9dc" exitCode=0
Oct 30 22:19:07.420860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.420805    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"
Oct 30 22:19:07.420908 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.420833    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" event={"ID":"a57febb1-62d4-475d-8a12-61cad4e3b93a","Type":"ContainerDied","Data":"2befb8b0a81b279f2fb31c1bc180faa0e30de8d22cd5d4258134cf6e3119d9dc"}
Oct 30 22:19:07.420952 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.420934    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp" event={"ID":"a57febb1-62d4-475d-8a12-61cad4e3b93a","Type":"ContainerDied","Data":"0ceb10c9e511052b40d79995d27d0ca06bfb6ebb3f3152b4415d13b7535864ba"}
Oct 30 22:19:07.421007 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.420969    4553 scope.go:122] "RemoveContainer" containerID="2befb8b0a81b279f2fb31c1bc180faa0e30de8d22cd5d4258134cf6e3119d9dc"
Oct 30 22:19:07.444402 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.444379    4553 scope.go:122] "RemoveContainer" containerID="2befb8b0a81b279f2fb31c1bc180faa0e30de8d22cd5d4258134cf6e3119d9dc"
Oct 30 22:19:07.444717 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:19:07.444700    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2befb8b0a81b279f2fb31c1bc180faa0e30de8d22cd5d4258134cf6e3119d9dc\": container with ID starting with 2befb8b0a81b279f2fb31c1bc180faa0e30de8d22cd5d4258134cf6e3119d9dc not found: ID does not exist" containerID="2befb8b0a81b279f2fb31c1bc180faa0e30de8d22cd5d4258134cf6e3119d9dc"
Oct 30 22:19:07.444767 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.444726    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2befb8b0a81b279f2fb31c1bc180faa0e30de8d22cd5d4258134cf6e3119d9dc"} err="failed to get container status \"2befb8b0a81b279f2fb31c1bc180faa0e30de8d22cd5d4258134cf6e3119d9dc\": rpc error: code = NotFound desc = could not find container \"2befb8b0a81b279f2fb31c1bc180faa0e30de8d22cd5d4258134cf6e3119d9dc\": container with ID starting with 2befb8b0a81b279f2fb31c1bc180faa0e30de8d22cd5d4258134cf6e3119d9dc not found: ID does not exist"
Oct 30 22:19:07.516315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.516268    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:07.516315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.516297    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbxft\" (UniqueName: \"kubernetes.io/projected/a57febb1-62d4-475d-8a12-61cad4e3b93a-kube-api-access-lbxft\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:07.516315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.516313    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:07.516315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.516327    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a57febb1-62d4-475d-8a12-61cad4e3b93a-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:07.631935 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.631888    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"]
Oct 30 22:19:07.635207 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:07.635183    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-mr2rp"]
Oct 30 22:19:09.395076 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:09.394991    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" path="/var/lib/kubelet/pods/a57febb1-62d4-475d-8a12-61cad4e3b93a/volumes"
Oct 30 22:19:14.301712 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:14.301674    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"]
Oct 30 22:19:14.302188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:14.301900    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7" podUID="b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" containerName="istio-proxy" containerID="cri-o://d7deea260d6770278fd6bbede79a18a8d2152eb18a0781bfe33b6ea5803ad8d1" gracePeriod=30
Oct 30 22:19:15.667221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:15.667172    4553 patch_prober.go:29] interesting pod/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:19:15.667652 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:15.667241    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7" podUID="b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:19:19.506997 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.506959    4553 generic.go:356] "Generic (PLEG): container finished" podID="b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" containerID="d7deea260d6770278fd6bbede79a18a8d2152eb18a0781bfe33b6ea5803ad8d1" exitCode=0
Oct 30 22:19:19.507452 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.507031    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7" event={"ID":"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c","Type":"ContainerDied","Data":"d7deea260d6770278fd6bbede79a18a8d2152eb18a0781bfe33b6ea5803ad8d1"}
Oct 30 22:19:19.579680 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.579619    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:19:19.676734 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.676699    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-podinfo\") pod \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") "
Oct 30 22:19:19.676943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.676785    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-data\") pod \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") "
Oct 30 22:19:19.676943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.676816    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-token\" (UniqueName: \"kubernetes.io/projected/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-token\") pod \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") "
Oct 30 22:19:19.676943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.676860    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-workload-socket\") pod \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") "
Oct 30 22:19:19.676943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.676882    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-credential-socket\") pod \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") "
Oct 30 22:19:19.676943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.676929    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istiod-ca-cert\") pod \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") "
Oct 30 22:19:19.677267 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.677029    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-kube-api-access-5gxtj\" (UniqueName: \"kubernetes.io/projected/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-kube-api-access-5gxtj\") pod \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") "
Oct 30 22:19:19.677267 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.677058    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-envoy\") pod \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") "
Oct 30 22:19:19.677267 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.677085    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-workload-certs\") pod \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\" (UID: \"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c\") "
Oct 30 22:19:19.677267 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.677192    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-data" pod "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" (UID: "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:19.677267 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.677234    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-workload-socket" pod "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" (UID: "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:19.677267 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.677246    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-credential-socket" pod "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" (UID: "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:19.677586 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.677418    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istiod-ca-cert" pod "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" (UID: "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:19.677586 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.677525    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-workload-certs" pod "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" (UID: "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:19.677859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.677686    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:19.677859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.677703    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:19.677859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.677713    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:19.677859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.677721    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:19.677859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.677730    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:19.679556 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.679529    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-token" pod "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" (UID: "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:19.679675 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.679532    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-envoy" pod "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" (UID: "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:19.679853 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.679775    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-podinfo" pod "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" (UID: "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:19:19.679853 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.679803    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-kube-api-access-5gxtj" pod "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" (UID: "b4c4dec5-f093-42e7-a8b1-68b5f2603a0c"). InnerVolumeSpecName "kube-api-access-5gxtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:19.778906 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.778871    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:19.778906 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.778907    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gxtj\" (UniqueName: \"kubernetes.io/projected/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-kube-api-access-5gxtj\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:19.778906 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.778925    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:19.778906 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:19.778938    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:20.514797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:20.514770    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"
Oct 30 22:19:20.515323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:20.514789    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7" event={"ID":"b4c4dec5-f093-42e7-a8b1-68b5f2603a0c","Type":"ContainerDied","Data":"5ff7f653c049788c128ce17e0de6b87b03236425d525327e45909420a540d9d5"}
Oct 30 22:19:20.515323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:20.514888    4553 scope.go:122] "RemoveContainer" containerID="d7deea260d6770278fd6bbede79a18a8d2152eb18a0781bfe33b6ea5803ad8d1"
Oct 30 22:19:20.734271 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:20.734233    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"]
Oct 30 22:19:20.783483 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:20.783391    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-hn9m7"]
Oct 30 22:19:21.393734 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:21.393700    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" path="/var/lib/kubelet/pods/b4c4dec5-f093-42e7-a8b1-68b5f2603a0c/volumes"
Oct 30 22:19:28.204251 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:19:28.204210    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:19:31.121330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.121289    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"]
Oct 30 22:19:31.121924 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.121512    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x" podUID="d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc" containerName="payload-processing" containerID="cri-o://afe3cbf5acc746413616083699d5e207b7f7a816df902afc11f9f42caec90f3a" gracePeriod=30
Oct 30 22:19:31.175860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.175824    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"]
Oct 30 22:19:31.176052 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.176034    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf" podUID="4be56767-e095-44c3-9a5b-5952102959cb" containerName="payload-pre-processing" containerID="cri-o://d0cff845328c9427daeabb0283305ee9292ab99a70cb28781c903aebec8be910" gracePeriod=30
Oct 30 22:19:31.470744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.470712    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"
Oct 30 22:19:31.491368 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.491340    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"
Oct 30 22:19:31.573437 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.573408    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be56767-e095-44c3-9a5b-5952102959cb-kube-api-access-g54db\" (UniqueName: \"kubernetes.io/projected/4be56767-e095-44c3-9a5b-5952102959cb-kube-api-access-g54db\") pod \"4be56767-e095-44c3-9a5b-5952102959cb\" (UID: \"4be56767-e095-44c3-9a5b-5952102959cb\") "
Oct 30 22:19:31.573599 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.573494    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc-kube-api-access-mqnpc\" (UniqueName: \"kubernetes.io/projected/d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc-kube-api-access-mqnpc\") pod \"d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc\" (UID: \"d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc\") "
Oct 30 22:19:31.573599 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.573530    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc-plugins-config-volume\") pod \"d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc\" (UID: \"d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc\") "
Oct 30 22:19:31.573674 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.573616    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be56767-e095-44c3-9a5b-5952102959cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be56767-e095-44c3-9a5b-5952102959cb-plugins-config-volume\") pod \"4be56767-e095-44c3-9a5b-5952102959cb\" (UID: \"4be56767-e095-44c3-9a5b-5952102959cb\") "
Oct 30 22:19:31.573993 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.573965    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc-plugins-config-volume" pod "d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc" (UID: "d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:31.573993 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.573985    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be56767-e095-44c3-9a5b-5952102959cb-plugins-config-volume" pod "4be56767-e095-44c3-9a5b-5952102959cb" (UID: "4be56767-e095-44c3-9a5b-5952102959cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:31.574307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.574151    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be56767-e095-44c3-9a5b-5952102959cb-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:31.574307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.574175    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:31.575771 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.575744    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be56767-e095-44c3-9a5b-5952102959cb-kube-api-access-g54db" pod "4be56767-e095-44c3-9a5b-5952102959cb" (UID: "4be56767-e095-44c3-9a5b-5952102959cb"). InnerVolumeSpecName "kube-api-access-g54db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:31.575879 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.575794    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc-kube-api-access-mqnpc" pod "d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc" (UID: "d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc"). InnerVolumeSpecName "kube-api-access-mqnpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:31.595039 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.595006    4553 generic.go:356] "Generic (PLEG): container finished" podID="d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc" containerID="afe3cbf5acc746413616083699d5e207b7f7a816df902afc11f9f42caec90f3a" exitCode=0
Oct 30 22:19:31.595219 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.595053    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"
Oct 30 22:19:31.595219 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.595086    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x" event={"ID":"d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc","Type":"ContainerDied","Data":"afe3cbf5acc746413616083699d5e207b7f7a816df902afc11f9f42caec90f3a"}
Oct 30 22:19:31.595219 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.595156    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x" event={"ID":"d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc","Type":"ContainerDied","Data":"7e74005671113a6c5ebc9f656acf2f329202f801daf0071abffdcb3aaefe870f"}
Oct 30 22:19:31.595219 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.595186    4553 scope.go:122] "RemoveContainer" containerID="afe3cbf5acc746413616083699d5e207b7f7a816df902afc11f9f42caec90f3a"
Oct 30 22:19:31.597111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.597084    4553 generic.go:356] "Generic (PLEG): container finished" podID="4be56767-e095-44c3-9a5b-5952102959cb" containerID="d0cff845328c9427daeabb0283305ee9292ab99a70cb28781c903aebec8be910" exitCode=0
Oct 30 22:19:31.597238 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.597180    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf" event={"ID":"4be56767-e095-44c3-9a5b-5952102959cb","Type":"ContainerDied","Data":"d0cff845328c9427daeabb0283305ee9292ab99a70cb28781c903aebec8be910"}
Oct 30 22:19:31.597238 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.597231    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf" event={"ID":"4be56767-e095-44c3-9a5b-5952102959cb","Type":"ContainerDied","Data":"f3661c4aae0d278146e4af819378fdabd401a9f34f1d59918ef8e68c9615dee6"}
Oct 30 22:19:31.597330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.597187    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"
Oct 30 22:19:31.610730 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.608009    4553 scope.go:122] "RemoveContainer" containerID="afe3cbf5acc746413616083699d5e207b7f7a816df902afc11f9f42caec90f3a"
Oct 30 22:19:31.610730 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:19:31.610599    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe3cbf5acc746413616083699d5e207b7f7a816df902afc11f9f42caec90f3a\": container with ID starting with afe3cbf5acc746413616083699d5e207b7f7a816df902afc11f9f42caec90f3a not found: ID does not exist" containerID="afe3cbf5acc746413616083699d5e207b7f7a816df902afc11f9f42caec90f3a"
Oct 30 22:19:31.610730 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.610649    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe3cbf5acc746413616083699d5e207b7f7a816df902afc11f9f42caec90f3a"} err="failed to get container status \"afe3cbf5acc746413616083699d5e207b7f7a816df902afc11f9f42caec90f3a\": rpc error: code = NotFound desc = could not find container \"afe3cbf5acc746413616083699d5e207b7f7a816df902afc11f9f42caec90f3a\": container with ID starting with afe3cbf5acc746413616083699d5e207b7f7a816df902afc11f9f42caec90f3a not found: ID does not exist"
Oct 30 22:19:31.610730 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.610677    4553 scope.go:122] "RemoveContainer" containerID="d0cff845328c9427daeabb0283305ee9292ab99a70cb28781c903aebec8be910"
Oct 30 22:19:31.708210 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.698703    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g54db\" (UniqueName: \"kubernetes.io/projected/4be56767-e095-44c3-9a5b-5952102959cb-kube-api-access-g54db\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:31.708210 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.698752    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqnpc\" (UniqueName: \"kubernetes.io/projected/d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc-kube-api-access-mqnpc\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:31.762933 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.762869    4553 scope.go:122] "RemoveContainer" containerID="d0cff845328c9427daeabb0283305ee9292ab99a70cb28781c903aebec8be910"
Oct 30 22:19:31.771177 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:19:31.767616    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0cff845328c9427daeabb0283305ee9292ab99a70cb28781c903aebec8be910\": container with ID starting with d0cff845328c9427daeabb0283305ee9292ab99a70cb28781c903aebec8be910 not found: ID does not exist" containerID="d0cff845328c9427daeabb0283305ee9292ab99a70cb28781c903aebec8be910"
Oct 30 22:19:31.771177 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.767666    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0cff845328c9427daeabb0283305ee9292ab99a70cb28781c903aebec8be910"} err="failed to get container status \"d0cff845328c9427daeabb0283305ee9292ab99a70cb28781c903aebec8be910\": rpc error: code = NotFound desc = could not find container \"d0cff845328c9427daeabb0283305ee9292ab99a70cb28781c903aebec8be910\": container with ID starting with d0cff845328c9427daeabb0283305ee9292ab99a70cb28781c903aebec8be910 not found: ID does not exist"
Oct 30 22:19:31.859876 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.859836    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"]
Oct 30 22:19:31.865125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.865093    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7a54db-5bd6ff8dd4-b9w5x"]
Oct 30 22:19:31.979033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.978807    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"]
Oct 30 22:19:31.982793 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:31.982768    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7a54db-6695679ff9-l6kwf"]
Oct 30 22:19:33.393376 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:33.393335    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be56767-e095-44c3-9a5b-5952102959cb" path="/var/lib/kubelet/pods/4be56767-e095-44c3-9a5b-5952102959cb/volumes"
Oct 30 22:19:33.393990 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:33.393968    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc" path="/var/lib/kubelet/pods/d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc/volumes"
Oct 30 22:19:39.909596 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:39.909559    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"]
Oct 30 22:19:39.910035 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:39.909778    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn" podUID="38824b8a-8c76-412c-9d99-eb8955f8d237" containerName="istio-proxy" containerID="cri-o://6969faaf03f1d297906e16cd281e15ce4db5ba823baa32afd20deeb28d2c5ab8" gracePeriod=30
Oct 30 22:19:42.232068 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:42.232026    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"]
Oct 30 22:19:42.343968 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:42.343931    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"]
Oct 30 22:19:43.773783 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:43.773337    4553 patch_prober.go:29] interesting pod/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:19:43.773783 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:43.773419    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn" podUID="38824b8a-8c76-412c-9d99-eb8955f8d237" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:19:45.175896 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.175827    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:19:45.205491 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.205461    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-workload-certs\") pod \"38824b8a-8c76-412c-9d99-eb8955f8d237\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") "
Oct 30 22:19:45.205673 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.205569    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-envoy\") pod \"38824b8a-8c76-412c-9d99-eb8955f8d237\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") "
Oct 30 22:19:45.205673 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.205648    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-workload-socket\") pod \"38824b8a-8c76-412c-9d99-eb8955f8d237\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") "
Oct 30 22:19:45.205813 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.205695    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-podinfo\") pod \"38824b8a-8c76-412c-9d99-eb8955f8d237\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") "
Oct 30 22:19:45.205813 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.205741    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38824b8a-8c76-412c-9d99-eb8955f8d237-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38824b8a-8c76-412c-9d99-eb8955f8d237-istiod-ca-cert\") pod \"38824b8a-8c76-412c-9d99-eb8955f8d237\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") "
Oct 30 22:19:45.205813 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.205738    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-workload-certs" pod "38824b8a-8c76-412c-9d99-eb8955f8d237" (UID: "38824b8a-8c76-412c-9d99-eb8955f8d237"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:45.205813 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.205777    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38824b8a-8c76-412c-9d99-eb8955f8d237-kube-api-access-7rd5l\" (UniqueName: \"kubernetes.io/projected/38824b8a-8c76-412c-9d99-eb8955f8d237-kube-api-access-7rd5l\") pod \"38824b8a-8c76-412c-9d99-eb8955f8d237\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") "
Oct 30 22:19:45.205813 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.205807    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-token\" (UniqueName: \"kubernetes.io/projected/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-token\") pod \"38824b8a-8c76-412c-9d99-eb8955f8d237\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") "
Oct 30 22:19:45.206054 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.205923    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-data\") pod \"38824b8a-8c76-412c-9d99-eb8955f8d237\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") "
Oct 30 22:19:45.206054 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.205941    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-workload-socket" pod "38824b8a-8c76-412c-9d99-eb8955f8d237" (UID: "38824b8a-8c76-412c-9d99-eb8955f8d237"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:45.206054 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.206001    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-credential-socket\") pod \"38824b8a-8c76-412c-9d99-eb8955f8d237\" (UID: \"38824b8a-8c76-412c-9d99-eb8955f8d237\") "
Oct 30 22:19:45.206231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.206115    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38824b8a-8c76-412c-9d99-eb8955f8d237-istiod-ca-cert" pod "38824b8a-8c76-412c-9d99-eb8955f8d237" (UID: "38824b8a-8c76-412c-9d99-eb8955f8d237"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:45.206286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.206234    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-data" pod "38824b8a-8c76-412c-9d99-eb8955f8d237" (UID: "38824b8a-8c76-412c-9d99-eb8955f8d237"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:45.206488 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.206409    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-credential-socket" pod "38824b8a-8c76-412c-9d99-eb8955f8d237" (UID: "38824b8a-8c76-412c-9d99-eb8955f8d237"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:45.206740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.206528    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:45.206740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.206543    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38824b8a-8c76-412c-9d99-eb8955f8d237-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:45.206740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.206552    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:45.206740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.206561    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:45.206740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.206570    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:45.208452 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.208426    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-token" pod "38824b8a-8c76-412c-9d99-eb8955f8d237" (UID: "38824b8a-8c76-412c-9d99-eb8955f8d237"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:45.208571 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.208498    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-podinfo" pod "38824b8a-8c76-412c-9d99-eb8955f8d237" (UID: "38824b8a-8c76-412c-9d99-eb8955f8d237"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:19:45.208681 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.208566    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38824b8a-8c76-412c-9d99-eb8955f8d237-kube-api-access-7rd5l" pod "38824b8a-8c76-412c-9d99-eb8955f8d237" (UID: "38824b8a-8c76-412c-9d99-eb8955f8d237"). InnerVolumeSpecName "kube-api-access-7rd5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:45.208681 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.208573    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-envoy" pod "38824b8a-8c76-412c-9d99-eb8955f8d237" (UID: "38824b8a-8c76-412c-9d99-eb8955f8d237"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:45.308194 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.308158    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:45.308194 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.308181    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:45.308194 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.308191    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rd5l\" (UniqueName: \"kubernetes.io/projected/38824b8a-8c76-412c-9d99-eb8955f8d237-kube-api-access-7rd5l\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:45.308194 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.308202    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38824b8a-8c76-412c-9d99-eb8955f8d237-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:45.765080 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.765041    4553 generic.go:356] "Generic (PLEG): container finished" podID="38824b8a-8c76-412c-9d99-eb8955f8d237" containerID="6969faaf03f1d297906e16cd281e15ce4db5ba823baa32afd20deeb28d2c5ab8" exitCode=0
Oct 30 22:19:45.765080 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.765077    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn" event={"ID":"38824b8a-8c76-412c-9d99-eb8955f8d237","Type":"ContainerDied","Data":"6969faaf03f1d297906e16cd281e15ce4db5ba823baa32afd20deeb28d2c5ab8"}
Oct 30 22:19:45.765404 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.765113    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"
Oct 30 22:19:45.765404 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.765152    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn" event={"ID":"38824b8a-8c76-412c-9d99-eb8955f8d237","Type":"ContainerDied","Data":"61beb54e3a85bcd2c8747c492cec9133ae65d38531d38e5e308b424b7f91ef85"}
Oct 30 22:19:45.765404 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.765188    4553 scope.go:122] "RemoveContainer" containerID="6969faaf03f1d297906e16cd281e15ce4db5ba823baa32afd20deeb28d2c5ab8"
Oct 30 22:19:45.778912 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.778889    4553 scope.go:122] "RemoveContainer" containerID="7b6b263643855f0afd3042400b79885698766cea56fed4bc4c929c90faee8d22"
Oct 30 22:19:45.788060 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.788039    4553 scope.go:122] "RemoveContainer" containerID="6969faaf03f1d297906e16cd281e15ce4db5ba823baa32afd20deeb28d2c5ab8"
Oct 30 22:19:45.788358 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:19:45.788337    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6969faaf03f1d297906e16cd281e15ce4db5ba823baa32afd20deeb28d2c5ab8\": container with ID starting with 6969faaf03f1d297906e16cd281e15ce4db5ba823baa32afd20deeb28d2c5ab8 not found: ID does not exist" containerID="6969faaf03f1d297906e16cd281e15ce4db5ba823baa32afd20deeb28d2c5ab8"
Oct 30 22:19:45.788458 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.788371    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6969faaf03f1d297906e16cd281e15ce4db5ba823baa32afd20deeb28d2c5ab8"} err="failed to get container status \"6969faaf03f1d297906e16cd281e15ce4db5ba823baa32afd20deeb28d2c5ab8\": rpc error: code = NotFound desc = could not find container \"6969faaf03f1d297906e16cd281e15ce4db5ba823baa32afd20deeb28d2c5ab8\": container with ID starting with 6969faaf03f1d297906e16cd281e15ce4db5ba823baa32afd20deeb28d2c5ab8 not found: ID does not exist"
Oct 30 22:19:45.788458 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.788395    4553 scope.go:122] "RemoveContainer" containerID="7b6b263643855f0afd3042400b79885698766cea56fed4bc4c929c90faee8d22"
Oct 30 22:19:45.788660 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:19:45.788639    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6b263643855f0afd3042400b79885698766cea56fed4bc4c929c90faee8d22\": container with ID starting with 7b6b263643855f0afd3042400b79885698766cea56fed4bc4c929c90faee8d22 not found: ID does not exist" containerID="7b6b263643855f0afd3042400b79885698766cea56fed4bc4c929c90faee8d22"
Oct 30 22:19:45.788699 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.788668    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6b263643855f0afd3042400b79885698766cea56fed4bc4c929c90faee8d22"} err="failed to get container status \"7b6b263643855f0afd3042400b79885698766cea56fed4bc4c929c90faee8d22\": rpc error: code = NotFound desc = could not find container \"7b6b263643855f0afd3042400b79885698766cea56fed4bc4c929c90faee8d22\": container with ID starting with 7b6b263643855f0afd3042400b79885698766cea56fed4bc4c929c90faee8d22 not found: ID does not exist"
Oct 30 22:19:45.881661 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.881635    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-hg9wk_05d7db25-7800-4217-a2fb-4396ba266c73/payload-processing/0.log"
Oct 30 22:19:45.932996 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.932963    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"]
Oct 30 22:19:45.935647 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:45.935626    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7a54db-openshift-default-68666f54f-9tggn"]
Oct 30 22:19:46.146690 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:46.146664    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-hg9wk_05d7db25-7800-4217-a2fb-4396ba266c73/payload-processing/0.log"
Oct 30 22:19:47.232871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.232821    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk" podUID="4190a170-7787-4d84-ac60-dfcfc0f33e32" containerName="payload-processing" containerID="cri-o://aa9dda03ac8a945ae0bc01548de226ee69781eed3eb0238ebeff8cbc72d2a33d" gracePeriod=25
Oct 30 22:19:47.344992 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.344952    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g" podUID="ffcd268a-e725-4a11-9c1d-1e27977c7b22" containerName="payload-pre-processing" containerID="cri-o://4d159233fbc4b83e81b073fcab394183dfc2ecdab3af03c91fbf07863cc2166b" gracePeriod=25
Oct 30 22:19:47.395529 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.395495    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38824b8a-8c76-412c-9d99-eb8955f8d237" path="/var/lib/kubelet/pods/38824b8a-8c76-412c-9d99-eb8955f8d237/volumes"
Oct 30 22:19:47.535415 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.535359    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"
Oct 30 22:19:47.637237 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.637210    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4190a170-7787-4d84-ac60-dfcfc0f33e32-kube-api-access-c8bcj\" (UniqueName: \"kubernetes.io/projected/4190a170-7787-4d84-ac60-dfcfc0f33e32-kube-api-access-c8bcj\") pod \"4190a170-7787-4d84-ac60-dfcfc0f33e32\" (UID: \"4190a170-7787-4d84-ac60-dfcfc0f33e32\") "
Oct 30 22:19:47.637429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.637374    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4190a170-7787-4d84-ac60-dfcfc0f33e32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4190a170-7787-4d84-ac60-dfcfc0f33e32-plugins-config-volume\") pod \"4190a170-7787-4d84-ac60-dfcfc0f33e32\" (UID: \"4190a170-7787-4d84-ac60-dfcfc0f33e32\") "
Oct 30 22:19:47.637739 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.637713    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4190a170-7787-4d84-ac60-dfcfc0f33e32-plugins-config-volume" pod "4190a170-7787-4d84-ac60-dfcfc0f33e32" (UID: "4190a170-7787-4d84-ac60-dfcfc0f33e32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:47.637867 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.637851    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4190a170-7787-4d84-ac60-dfcfc0f33e32-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:47.639499 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.639478    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4190a170-7787-4d84-ac60-dfcfc0f33e32-kube-api-access-c8bcj" pod "4190a170-7787-4d84-ac60-dfcfc0f33e32" (UID: "4190a170-7787-4d84-ac60-dfcfc0f33e32"). InnerVolumeSpecName "kube-api-access-c8bcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:47.644195 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.644175    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"
Oct 30 22:19:47.738991 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.738905    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffcd268a-e725-4a11-9c1d-1e27977c7b22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcd268a-e725-4a11-9c1d-1e27977c7b22-plugins-config-volume\") pod \"ffcd268a-e725-4a11-9c1d-1e27977c7b22\" (UID: \"ffcd268a-e725-4a11-9c1d-1e27977c7b22\") "
Oct 30 22:19:47.738991 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.738948    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcd268a-e725-4a11-9c1d-1e27977c7b22-kube-api-access-8q7ts\" (UniqueName: \"kubernetes.io/projected/ffcd268a-e725-4a11-9c1d-1e27977c7b22-kube-api-access-8q7ts\") pod \"ffcd268a-e725-4a11-9c1d-1e27977c7b22\" (UID: \"ffcd268a-e725-4a11-9c1d-1e27977c7b22\") "
Oct 30 22:19:47.739299 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.739277    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcd268a-e725-4a11-9c1d-1e27977c7b22-plugins-config-volume" pod "ffcd268a-e725-4a11-9c1d-1e27977c7b22" (UID: "ffcd268a-e725-4a11-9c1d-1e27977c7b22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:47.739535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.739521    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffcd268a-e725-4a11-9c1d-1e27977c7b22-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:47.739598 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.739537    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8bcj\" (UniqueName: \"kubernetes.io/projected/4190a170-7787-4d84-ac60-dfcfc0f33e32-kube-api-access-c8bcj\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:47.741344 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.741319    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcd268a-e725-4a11-9c1d-1e27977c7b22-kube-api-access-8q7ts" pod "ffcd268a-e725-4a11-9c1d-1e27977c7b22" (UID: "ffcd268a-e725-4a11-9c1d-1e27977c7b22"). InnerVolumeSpecName "kube-api-access-8q7ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:47.782822 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.782792    4553 generic.go:356] "Generic (PLEG): container finished" podID="4190a170-7787-4d84-ac60-dfcfc0f33e32" containerID="aa9dda03ac8a945ae0bc01548de226ee69781eed3eb0238ebeff8cbc72d2a33d" exitCode=0
Oct 30 22:19:47.783003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.782841    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"
Oct 30 22:19:47.783003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.782870    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk" event={"ID":"4190a170-7787-4d84-ac60-dfcfc0f33e32","Type":"ContainerDied","Data":"aa9dda03ac8a945ae0bc01548de226ee69781eed3eb0238ebeff8cbc72d2a33d"}
Oct 30 22:19:47.783003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.782928    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk" event={"ID":"4190a170-7787-4d84-ac60-dfcfc0f33e32","Type":"ContainerDied","Data":"d40b4eb06f4a0b0dee2173b8986df0da98a160f9f4018646affe601606924d0e"}
Oct 30 22:19:47.783003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.782973    4553 scope.go:122] "RemoveContainer" containerID="aa9dda03ac8a945ae0bc01548de226ee69781eed3eb0238ebeff8cbc72d2a33d"
Oct 30 22:19:47.785406 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.785212    4553 generic.go:356] "Generic (PLEG): container finished" podID="ffcd268a-e725-4a11-9c1d-1e27977c7b22" containerID="4d159233fbc4b83e81b073fcab394183dfc2ecdab3af03c91fbf07863cc2166b" exitCode=0
Oct 30 22:19:47.785406 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.785332    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g" event={"ID":"ffcd268a-e725-4a11-9c1d-1e27977c7b22","Type":"ContainerDied","Data":"4d159233fbc4b83e81b073fcab394183dfc2ecdab3af03c91fbf07863cc2166b"}
Oct 30 22:19:47.785406 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.785355    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"
Oct 30 22:19:47.785406 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.785373    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g" event={"ID":"ffcd268a-e725-4a11-9c1d-1e27977c7b22","Type":"ContainerDied","Data":"1ed3eae21c9bbe09f925763148663ebb78a5e11e50d80388a0b1f693a216b280"}
Oct 30 22:19:47.795814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.795773    4553 scope.go:122] "RemoveContainer" containerID="aa9dda03ac8a945ae0bc01548de226ee69781eed3eb0238ebeff8cbc72d2a33d"
Oct 30 22:19:47.796257 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:19:47.796188    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9dda03ac8a945ae0bc01548de226ee69781eed3eb0238ebeff8cbc72d2a33d\": container with ID starting with aa9dda03ac8a945ae0bc01548de226ee69781eed3eb0238ebeff8cbc72d2a33d not found: ID does not exist" containerID="aa9dda03ac8a945ae0bc01548de226ee69781eed3eb0238ebeff8cbc72d2a33d"
Oct 30 22:19:47.796257 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.796228    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9dda03ac8a945ae0bc01548de226ee69781eed3eb0238ebeff8cbc72d2a33d"} err="failed to get container status \"aa9dda03ac8a945ae0bc01548de226ee69781eed3eb0238ebeff8cbc72d2a33d\": rpc error: code = NotFound desc = could not find container \"aa9dda03ac8a945ae0bc01548de226ee69781eed3eb0238ebeff8cbc72d2a33d\": container with ID starting with aa9dda03ac8a945ae0bc01548de226ee69781eed3eb0238ebeff8cbc72d2a33d not found: ID does not exist"
Oct 30 22:19:47.796257 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.796254    4553 scope.go:122] "RemoveContainer" containerID="4d159233fbc4b83e81b073fcab394183dfc2ecdab3af03c91fbf07863cc2166b"
Oct 30 22:19:47.811101 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.811037    4553 scope.go:122] "RemoveContainer" containerID="4d159233fbc4b83e81b073fcab394183dfc2ecdab3af03c91fbf07863cc2166b"
Oct 30 22:19:47.811617 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:19:47.811518    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d159233fbc4b83e81b073fcab394183dfc2ecdab3af03c91fbf07863cc2166b\": container with ID starting with 4d159233fbc4b83e81b073fcab394183dfc2ecdab3af03c91fbf07863cc2166b not found: ID does not exist" containerID="4d159233fbc4b83e81b073fcab394183dfc2ecdab3af03c91fbf07863cc2166b"
Oct 30 22:19:47.811617 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.811559    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d159233fbc4b83e81b073fcab394183dfc2ecdab3af03c91fbf07863cc2166b"} err="failed to get container status \"4d159233fbc4b83e81b073fcab394183dfc2ecdab3af03c91fbf07863cc2166b\": rpc error: code = NotFound desc = could not find container \"4d159233fbc4b83e81b073fcab394183dfc2ecdab3af03c91fbf07863cc2166b\": container with ID starting with 4d159233fbc4b83e81b073fcab394183dfc2ecdab3af03c91fbf07863cc2166b not found: ID does not exist"
Oct 30 22:19:47.840641 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.840607    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q7ts\" (UniqueName: \"kubernetes.io/projected/ffcd268a-e725-4a11-9c1d-1e27977c7b22-kube-api-access-8q7ts\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:47.961060 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.961026    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"]
Oct 30 22:19:47.968919 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:47.968892    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-9847b887-f7m7g"]
Oct 30 22:19:48.079274 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:48.079186    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"]
Oct 30 22:19:48.081825 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:48.081802    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96567a-b8d48f6b7-w22gk"]
Oct 30 22:19:49.396318 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:49.396285    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4190a170-7787-4d84-ac60-dfcfc0f33e32" path="/var/lib/kubelet/pods/4190a170-7787-4d84-ac60-dfcfc0f33e32/volumes"
Oct 30 22:19:49.397518 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:49.397493    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffcd268a-e725-4a11-9c1d-1e27977c7b22" path="/var/lib/kubelet/pods/ffcd268a-e725-4a11-9c1d-1e27977c7b22/volumes"
Oct 30 22:19:55.194651 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.194617    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"]
Oct 30 22:19:55.195288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195273    4553 cpu_manager.go:414] "containerMap: removing container" podUID="38824b8a-8c76-412c-9d99-eb8955f8d237" containerName="istio-proxy"
Oct 30 22:19:55.195288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195289    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38824b8a-8c76-412c-9d99-eb8955f8d237" containerName="istio-proxy"
Oct 30 22:19:55.195393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195308    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerName="istio-proxy"
Oct 30 22:19:55.195393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195314    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerName="istio-proxy"
Oct 30 22:19:55.195393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195321    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc" containerName="payload-processing"
Oct 30 22:19:55.195393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195328    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc" containerName="payload-processing"
Oct 30 22:19:55.195393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195338    4553 cpu_manager.go:414] "containerMap: removing container" podUID="b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" containerName="istio-proxy"
Oct 30 22:19:55.195393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195343    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" containerName="istio-proxy"
Oct 30 22:19:55.195393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195369    4553 cpu_manager.go:414] "containerMap: removing container" podUID="4be56767-e095-44c3-9a5b-5952102959cb" containerName="payload-pre-processing"
Oct 30 22:19:55.195393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195374    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be56767-e095-44c3-9a5b-5952102959cb" containerName="payload-pre-processing"
Oct 30 22:19:55.195393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195383    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ffcd268a-e725-4a11-9c1d-1e27977c7b22" containerName="payload-pre-processing"
Oct 30 22:19:55.195393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195387    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffcd268a-e725-4a11-9c1d-1e27977c7b22" containerName="payload-pre-processing"
Oct 30 22:19:55.195393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195394    4553 cpu_manager.go:414] "containerMap: removing container" podUID="4190a170-7787-4d84-ac60-dfcfc0f33e32" containerName="payload-processing"
Oct 30 22:19:55.195393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195399    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4190a170-7787-4d84-ac60-dfcfc0f33e32" containerName="payload-processing"
Oct 30 22:19:55.195744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195473    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be56767-e095-44c3-9a5b-5952102959cb" containerName="payload-pre-processing"
Oct 30 22:19:55.195744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195481    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffcd268a-e725-4a11-9c1d-1e27977c7b22" containerName="payload-pre-processing"
Oct 30 22:19:55.195744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195487    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="38824b8a-8c76-412c-9d99-eb8955f8d237" containerName="istio-proxy"
Oct 30 22:19:55.195744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195493    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="4190a170-7787-4d84-ac60-dfcfc0f33e32" containerName="payload-processing"
Oct 30 22:19:55.195744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195503    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="a57febb1-62d4-475d-8a12-61cad4e3b93a" containerName="istio-proxy"
Oct 30 22:19:55.195744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195507    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d40c35-7fbd-4b56-9bd4-1f45b0b198cc" containerName="payload-processing"
Oct 30 22:19:55.195744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.195512    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c4dec5-f093-42e7-a8b1-68b5f2603a0c" containerName="istio-proxy"
Oct 30 22:19:55.202204 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.202167    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:55.255026 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.254987    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"]
Oct 30 22:19:55.255907 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.255884    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8b1e0a-serving-cert\""
Oct 30 22:19:55.334981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.334940    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-metrics-certs\") pod \"maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:55.335205 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.335155    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t7g8\" (UniqueName: \"kubernetes.io/projected/4276289b-13bb-4d36-8032-0d12341413b4-kube-api-access-2t7g8\") pod \"maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:55.335283 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.335265    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:55.335340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.335314    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4276289b-13bb-4d36-8032-0d12341413b4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:55.436944 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.436908    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t7g8\" (UniqueName: \"kubernetes.io/projected/4276289b-13bb-4d36-8032-0d12341413b4-kube-api-access-2t7g8\") pod \"maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:55.437177 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.436992    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:55.437177 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.437048    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4276289b-13bb-4d36-8032-0d12341413b4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:55.437327 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.437232    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-metrics-certs\") pod \"maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:55.437439 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:19:55.437381    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-serving-cert: secret "maas-api-e2e-ipp-a-8b1e0a-serving-cert" not found
Oct 30 22:19:55.437567 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:19:55.437497    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-maas-api-tls podName:4276289b-13bb-4d36-8032-0d12341413b4 nodeName:}" failed. No retries permitted until 2026-10-30 22:19:55.937474655 +0000 UTC m=+2246.671837664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-maas-api-tls") pod "maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq" (UID: "4276289b-13bb-4d36-8032-0d12341413b4") : secret "maas-api-e2e-ipp-a-8b1e0a-serving-cert" not found
Oct 30 22:19:55.438326 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.438305    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4276289b-13bb-4d36-8032-0d12341413b4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:55.471259 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.463736    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-metrics-certs\") pod \"maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:55.496057 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.496023    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t7g8\" (UniqueName: \"kubernetes.io/projected/4276289b-13bb-4d36-8032-0d12341413b4-kube-api-access-2t7g8\") pod \"maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:55.944464 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.944429    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:55.947174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:55.947120    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:56.121006 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:56.120973    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:56.501340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:56.501312    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"]
Oct 30 22:19:56.502124 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:19:56.502099    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4276289b_13bb_4d36_8032_0d12341413b4.slice/crio-83fc50bafc8c7e8cda919a875fe96eeae81e89adc7226a9d6709e3664129946f WatchSource:0}: Error finding container 83fc50bafc8c7e8cda919a875fe96eeae81e89adc7226a9d6709e3664129946f: Status 404 returned error can't find the container with id 83fc50bafc8c7e8cda919a875fe96eeae81e89adc7226a9d6709e3664129946f
Oct 30 22:19:56.849409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:56.849367    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq" event={"ID":"4276289b-13bb-4d36-8032-0d12341413b4","Type":"ContainerStarted","Data":"83fc50bafc8c7e8cda919a875fe96eeae81e89adc7226a9d6709e3664129946f"}
Oct 30 22:19:57.856742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:57.856708    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq" event={"ID":"4276289b-13bb-4d36-8032-0d12341413b4","Type":"ContainerStarted","Data":"04ed7525a89e1f736540bd064d76d8498d299edee879caf3dace0e3f51aa79cd"}
Oct 30 22:19:57.857195 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:57.856873    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:19:57.971903 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:19:57.971845    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq" podStartSLOduration=2.1569450039999998 podStartE2EDuration="2.971826484s" totalImagesPullingTime="814.88148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:57.971013698 +0000 UTC m=+2248.705376731" watchObservedRunningTime="2026-10-30 22:19:57.971826484 +0000 UTC m=+2248.706189516"
Oct 30 22:19:58.203861 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:19:58.203767    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:20:03.868359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:03.868324    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:20:16.955892 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:16.943657    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"]
Oct 30 22:20:16.955892 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:16.944666    4553 cpu_manager.go:414] "containerMap: removing container" podUID="38824b8a-8c76-412c-9d99-eb8955f8d237" containerName="istio-proxy"
Oct 30 22:20:16.955892 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:16.944688    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38824b8a-8c76-412c-9d99-eb8955f8d237" containerName="istio-proxy"
Oct 30 22:20:16.955892 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:16.944916    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="38824b8a-8c76-412c-9d99-eb8955f8d237" containerName="istio-proxy"
Oct 30 22:20:16.967310 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:16.967280    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"
Oct 30 22:20:17.047811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.033584    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"]
Oct 30 22:20:17.059524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.059473    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-fe2565-dockercfg-hqtss\""
Oct 30 22:20:17.064662 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.060873    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-fe2565\""
Oct 30 22:20:17.068053 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.067712    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6108731-ab6a-416d-909b-662dba26de39-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs\" (UID: \"a6108731-ab6a-416d-909b-662dba26de39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"
Oct 30 22:20:17.068270 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.068207    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmlcf\" (UniqueName: \"kubernetes.io/projected/a6108731-ab6a-416d-909b-662dba26de39-kube-api-access-rmlcf\") pod \"payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs\" (UID: \"a6108731-ab6a-416d-909b-662dba26de39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"
Oct 30 22:20:17.132023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.131989    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"]
Oct 30 22:20:17.136111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.136095    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"
Oct 30 22:20:17.169999 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.169969    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6108731-ab6a-416d-909b-662dba26de39-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs\" (UID: \"a6108731-ab6a-416d-909b-662dba26de39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"
Oct 30 22:20:17.170157 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.170034    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmlcf\" (UniqueName: \"kubernetes.io/projected/a6108731-ab6a-416d-909b-662dba26de39-kube-api-access-rmlcf\") pod \"payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs\" (UID: \"a6108731-ab6a-416d-909b-662dba26de39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"
Oct 30 22:20:17.170670 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.170652    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6108731-ab6a-416d-909b-662dba26de39-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs\" (UID: \"a6108731-ab6a-416d-909b-662dba26de39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"
Oct 30 22:20:17.198898 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.198865    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"]
Oct 30 22:20:17.232830 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.232723    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmlcf\" (UniqueName: \"kubernetes.io/projected/a6108731-ab6a-416d-909b-662dba26de39-kube-api-access-rmlcf\") pod \"payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs\" (UID: \"a6108731-ab6a-416d-909b-662dba26de39\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"
Oct 30 22:20:17.271562 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.271526    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx6gc\" (UniqueName: \"kubernetes.io/projected/3212edbe-040d-489b-8e26-7371e06cbacd-kube-api-access-mx6gc\") pod \"payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24\" (UID: \"3212edbe-040d-489b-8e26-7371e06cbacd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"
Oct 30 22:20:17.271762 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.271745    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3212edbe-040d-489b-8e26-7371e06cbacd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24\" (UID: \"3212edbe-040d-489b-8e26-7371e06cbacd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"
Oct 30 22:20:17.341984 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.341951    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"
Oct 30 22:20:17.372773 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.372728    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx6gc\" (UniqueName: \"kubernetes.io/projected/3212edbe-040d-489b-8e26-7371e06cbacd-kube-api-access-mx6gc\") pod \"payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24\" (UID: \"3212edbe-040d-489b-8e26-7371e06cbacd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"
Oct 30 22:20:17.372946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.372931    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3212edbe-040d-489b-8e26-7371e06cbacd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24\" (UID: \"3212edbe-040d-489b-8e26-7371e06cbacd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"
Oct 30 22:20:17.373913 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.373890    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3212edbe-040d-489b-8e26-7371e06cbacd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24\" (UID: \"3212edbe-040d-489b-8e26-7371e06cbacd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"
Oct 30 22:20:17.436699 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.436655    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx6gc\" (UniqueName: \"kubernetes.io/projected/3212edbe-040d-489b-8e26-7371e06cbacd-kube-api-access-mx6gc\") pod \"payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24\" (UID: \"3212edbe-040d-489b-8e26-7371e06cbacd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"
Oct 30 22:20:17.451198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.450470    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"
Oct 30 22:20:17.734670 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.734462    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"]
Oct 30 22:20:17.737997 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:20:17.737962    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6108731_ab6a_416d_909b_662dba26de39.slice/crio-4e8dca81ab3e915d3212dacbb83cea8ee36ef057423d79b95e1491cd683e54ec WatchSource:0}: Error finding container 4e8dca81ab3e915d3212dacbb83cea8ee36ef057423d79b95e1491cd683e54ec: Status 404 returned error can't find the container with id 4e8dca81ab3e915d3212dacbb83cea8ee36ef057423d79b95e1491cd683e54ec
Oct 30 22:20:17.840174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:17.840128    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"]
Oct 30 22:20:17.842231 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:20:17.842203    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3212edbe_040d_489b_8e26_7371e06cbacd.slice/crio-490eb588cfd97d0c6845e57047764f8880deb7b0b3bb586ded2dc19f1ff34c9a WatchSource:0}: Error finding container 490eb588cfd97d0c6845e57047764f8880deb7b0b3bb586ded2dc19f1ff34c9a: Status 404 returned error can't find the container with id 490eb588cfd97d0c6845e57047764f8880deb7b0b3bb586ded2dc19f1ff34c9a
Oct 30 22:20:18.011680 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:18.011646    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24" event={"ID":"3212edbe-040d-489b-8e26-7371e06cbacd","Type":"ContainerStarted","Data":"6e168c04f425124cf21f10a9bd3327d043110b155258a15c00e8b23f5af0d36d"}
Oct 30 22:20:18.012163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:18.011709    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24" event={"ID":"3212edbe-040d-489b-8e26-7371e06cbacd","Type":"ContainerStarted","Data":"490eb588cfd97d0c6845e57047764f8880deb7b0b3bb586ded2dc19f1ff34c9a"}
Oct 30 22:20:18.012163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:18.011741    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"
Oct 30 22:20:18.013700 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:18.013661    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs" event={"ID":"a6108731-ab6a-416d-909b-662dba26de39","Type":"ContainerStarted","Data":"f843d5f6b54043574c533ce72d5ee7c1b1d4bf83f21b9d9d50acd20d218aa328"}
Oct 30 22:20:18.013843 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:18.013717    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs" event={"ID":"a6108731-ab6a-416d-909b-662dba26de39","Type":"ContainerStarted","Data":"4e8dca81ab3e915d3212dacbb83cea8ee36ef057423d79b95e1491cd683e54ec"}
Oct 30 22:20:18.013843 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:18.013739    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"
Oct 30 22:20:18.129333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:18.129278    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24" podStartSLOduration=1.129260997 podStartE2EDuration="1.129260997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:18.127587448 +0000 UTC m=+2268.861950478" watchObservedRunningTime="2026-10-30 22:20:18.129260997 +0000 UTC m=+2268.863624067"
Oct 30 22:20:18.270809 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:18.265509    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs" podStartSLOduration=2.265487979 podStartE2EDuration="2.265487979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:18.262282695 +0000 UTC m=+2268.996645729" watchObservedRunningTime="2026-10-30 22:20:18.265487979 +0000 UTC m=+2268.999851011"
Oct 30 22:20:28.203975 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:20:28.203934    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:20:29.023203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:29.023169    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"
Oct 30 22:20:29.023414 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:29.023270    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"
Oct 30 22:20:32.776467 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:32.776426    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"]
Oct 30 22:20:32.782726 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:32.782699    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"
Oct 30 22:20:32.790786 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:32.790215    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmhrp\" (UniqueName: \"kubernetes.io/projected/dc594a39-569b-4782-8396-4074ac150c4a-kube-api-access-xmhrp\") pod \"payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh\" (UID: \"dc594a39-569b-4782-8396-4074ac150c4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"
Oct 30 22:20:32.790786 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:32.790320    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc594a39-569b-4782-8396-4074ac150c4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh\" (UID: \"dc594a39-569b-4782-8396-4074ac150c4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"
Oct 30 22:20:32.836542 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:32.836506    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"]
Oct 30 22:20:32.882821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:32.882788    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a28730\""
Oct 30 22:20:32.883013 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:32.882886    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a28730-dockercfg-2qn7z\""
Oct 30 22:20:32.891569 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:32.891530    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmhrp\" (UniqueName: \"kubernetes.io/projected/dc594a39-569b-4782-8396-4074ac150c4a-kube-api-access-xmhrp\") pod \"payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh\" (UID: \"dc594a39-569b-4782-8396-4074ac150c4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"
Oct 30 22:20:32.891748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:32.891592    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc594a39-569b-4782-8396-4074ac150c4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh\" (UID: \"dc594a39-569b-4782-8396-4074ac150c4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"
Oct 30 22:20:32.892360 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:32.892341    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc594a39-569b-4782-8396-4074ac150c4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh\" (UID: \"dc594a39-569b-4782-8396-4074ac150c4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"
Oct 30 22:20:32.950810 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:32.950765    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmhrp\" (UniqueName: \"kubernetes.io/projected/dc594a39-569b-4782-8396-4074ac150c4a-kube-api-access-xmhrp\") pod \"payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh\" (UID: \"dc594a39-569b-4782-8396-4074ac150c4a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"
Oct 30 22:20:33.097080 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:33.097052    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"
Oct 30 22:20:33.476248 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:33.476218    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"]
Oct 30 22:20:33.477219 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:20:33.477190    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc594a39_569b_4782_8396_4074ac150c4a.slice/crio-bb749cd91e80b7dc2fb77cf1c4de00f95a13d44ec75c343ffd9ee8f4aba6d422 WatchSource:0}: Error finding container bb749cd91e80b7dc2fb77cf1c4de00f95a13d44ec75c343ffd9ee8f4aba6d422: Status 404 returned error can't find the container with id bb749cd91e80b7dc2fb77cf1c4de00f95a13d44ec75c343ffd9ee8f4aba6d422
Oct 30 22:20:33.543768 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:33.543733    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"]
Oct 30 22:20:33.549156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:33.549117    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"
Oct 30 22:20:33.602513 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:33.602462    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv9j8\" (UniqueName: \"kubernetes.io/projected/09d3a76e-ef0c-4ec6-b8c4-322e180ad95a-kube-api-access-hv9j8\") pod \"payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg\" (UID: \"09d3a76e-ef0c-4ec6-b8c4-322e180ad95a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"
Oct 30 22:20:33.602722 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:33.602631    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d3a76e-ef0c-4ec6-b8c4-322e180ad95a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg\" (UID: \"09d3a76e-ef0c-4ec6-b8c4-322e180ad95a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"
Oct 30 22:20:33.605619 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:33.605591    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"]
Oct 30 22:20:33.704401 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:33.704359    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv9j8\" (UniqueName: \"kubernetes.io/projected/09d3a76e-ef0c-4ec6-b8c4-322e180ad95a-kube-api-access-hv9j8\") pod \"payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg\" (UID: \"09d3a76e-ef0c-4ec6-b8c4-322e180ad95a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"
Oct 30 22:20:33.704627 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:33.704434    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d3a76e-ef0c-4ec6-b8c4-322e180ad95a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg\" (UID: \"09d3a76e-ef0c-4ec6-b8c4-322e180ad95a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"
Oct 30 22:20:33.705451 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:33.705419    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d3a76e-ef0c-4ec6-b8c4-322e180ad95a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg\" (UID: \"09d3a76e-ef0c-4ec6-b8c4-322e180ad95a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"
Oct 30 22:20:33.762343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:33.762270    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv9j8\" (UniqueName: \"kubernetes.io/projected/09d3a76e-ef0c-4ec6-b8c4-322e180ad95a-kube-api-access-hv9j8\") pod \"payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg\" (UID: \"09d3a76e-ef0c-4ec6-b8c4-322e180ad95a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"
Oct 30 22:20:33.864207 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:33.864173    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"
Oct 30 22:20:34.153958 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:34.153920    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh" event={"ID":"dc594a39-569b-4782-8396-4074ac150c4a","Type":"ContainerStarted","Data":"5991767be767c2ff85cd218c2fd40faca559cf93f16cb1069182fd7ba99017dc"}
Oct 30 22:20:34.154210 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:34.153984    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh" event={"ID":"dc594a39-569b-4782-8396-4074ac150c4a","Type":"ContainerStarted","Data":"bb749cd91e80b7dc2fb77cf1c4de00f95a13d44ec75c343ffd9ee8f4aba6d422"}
Oct 30 22:20:34.154210 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:34.154013    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"
Oct 30 22:20:34.250482 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:20:34.250451    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d3a76e_ef0c_4ec6_b8c4_322e180ad95a.slice/crio-6203aa2f040fa638263402aeb0f5a30ef940585af8abbde2a6dc6e6720fb54f7 WatchSource:0}: Error finding container 6203aa2f040fa638263402aeb0f5a30ef940585af8abbde2a6dc6e6720fb54f7: Status 404 returned error can't find the container with id 6203aa2f040fa638263402aeb0f5a30ef940585af8abbde2a6dc6e6720fb54f7
Oct 30 22:20:34.250630 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:34.250573    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"]
Oct 30 22:20:34.269780 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:34.269738    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh" podStartSLOduration=2.269722688 podStartE2EDuration="2.269722688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:34.268606155 +0000 UTC m=+2285.002969191" watchObservedRunningTime="2026-10-30 22:20:34.269722688 +0000 UTC m=+2285.004085760"
Oct 30 22:20:35.161446 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:35.161403    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg" event={"ID":"09d3a76e-ef0c-4ec6-b8c4-322e180ad95a","Type":"ContainerStarted","Data":"30fe2729a68c9e7a83cda2c11404830c7eb612fa6089634b58aa8dc5496b3b1d"}
Oct 30 22:20:35.161854 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:35.161634    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"
Oct 30 22:20:35.161854 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:35.161665    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg" event={"ID":"09d3a76e-ef0c-4ec6-b8c4-322e180ad95a","Type":"ContainerStarted","Data":"6203aa2f040fa638263402aeb0f5a30ef940585af8abbde2a6dc6e6720fb54f7"}
Oct 30 22:20:35.280301 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:35.280251    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg" podStartSLOduration=2.280237164 podStartE2EDuration="2.280237164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:35.278162029 +0000 UTC m=+2286.012525065" watchObservedRunningTime="2026-10-30 22:20:35.280237164 +0000 UTC m=+2286.014600195"
Oct 30 22:20:45.163792 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:45.163755    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"
Oct 30 22:20:46.171109 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:46.171078    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"
Oct 30 22:20:51.305712 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:51.305677    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"]
Oct 30 22:20:51.311015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:51.310993    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"
Oct 30 22:20:51.368187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:51.368158    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"]
Oct 30 22:20:51.414702 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:51.414671    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-96567a-dockercfg-g6hvv\""
Oct 30 22:20:51.414878 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:51.414731    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-96567a\""
Oct 30 22:20:51.446802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:51.446758    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad305e1a-a602-4c0c-9454-f0d783e281fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g\" (UID: \"ad305e1a-a602-4c0c-9454-f0d783e281fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"
Oct 30 22:20:51.447111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:51.447088    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kvvp\" (UniqueName: \"kubernetes.io/projected/ad305e1a-a602-4c0c-9454-f0d783e281fb-kube-api-access-7kvvp\") pod \"payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g\" (UID: \"ad305e1a-a602-4c0c-9454-f0d783e281fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"
Oct 30 22:20:51.548779 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:51.548741    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kvvp\" (UniqueName: \"kubernetes.io/projected/ad305e1a-a602-4c0c-9454-f0d783e281fb-kube-api-access-7kvvp\") pod \"payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g\" (UID: \"ad305e1a-a602-4c0c-9454-f0d783e281fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"
Oct 30 22:20:51.548987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:51.548880    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad305e1a-a602-4c0c-9454-f0d783e281fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g\" (UID: \"ad305e1a-a602-4c0c-9454-f0d783e281fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"
Oct 30 22:20:51.549616 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:51.549597    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad305e1a-a602-4c0c-9454-f0d783e281fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g\" (UID: \"ad305e1a-a602-4c0c-9454-f0d783e281fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"
Oct 30 22:20:51.607766 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:51.607729    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kvvp\" (UniqueName: \"kubernetes.io/projected/ad305e1a-a602-4c0c-9454-f0d783e281fb-kube-api-access-7kvvp\") pod \"payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g\" (UID: \"ad305e1a-a602-4c0c-9454-f0d783e281fb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"
Oct 30 22:20:51.625199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:51.625166    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"
Oct 30 22:20:52.003403 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.003367    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"]
Oct 30 22:20:52.004967 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:20:52.004932    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad305e1a_a602_4c0c_9454_f0d783e281fb.slice/crio-cb2767ba87cb908cb5cebc4baa33d5881616047be606c4cacf7d9d6f420c373b WatchSource:0}: Error finding container cb2767ba87cb908cb5cebc4baa33d5881616047be606c4cacf7d9d6f420c373b: Status 404 returned error can't find the container with id cb2767ba87cb908cb5cebc4baa33d5881616047be606c4cacf7d9d6f420c373b
Oct 30 22:20:52.090545 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.090505    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"]
Oct 30 22:20:52.094784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.094761    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"
Oct 30 22:20:52.152445 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.152416    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"]
Oct 30 22:20:52.156667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.156638    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhvw5\" (UniqueName: \"kubernetes.io/projected/a37cba11-3e73-47d7-88c8-133f37788561-kube-api-access-rhvw5\") pod \"payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh\" (UID: \"a37cba11-3e73-47d7-88c8-133f37788561\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"
Oct 30 22:20:52.157041 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.157015    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37cba11-3e73-47d7-88c8-133f37788561-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh\" (UID: \"a37cba11-3e73-47d7-88c8-133f37788561\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"
Oct 30 22:20:52.258860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.258763    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvw5\" (UniqueName: \"kubernetes.io/projected/a37cba11-3e73-47d7-88c8-133f37788561-kube-api-access-rhvw5\") pod \"payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh\" (UID: \"a37cba11-3e73-47d7-88c8-133f37788561\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"
Oct 30 22:20:52.259038 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.258980    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37cba11-3e73-47d7-88c8-133f37788561-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh\" (UID: \"a37cba11-3e73-47d7-88c8-133f37788561\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"
Oct 30 22:20:52.259724 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.259696    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37cba11-3e73-47d7-88c8-133f37788561-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh\" (UID: \"a37cba11-3e73-47d7-88c8-133f37788561\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"
Oct 30 22:20:52.318796 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.318764    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvw5\" (UniqueName: \"kubernetes.io/projected/a37cba11-3e73-47d7-88c8-133f37788561-kube-api-access-rhvw5\") pod \"payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh\" (UID: \"a37cba11-3e73-47d7-88c8-133f37788561\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"
Oct 30 22:20:52.321887 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.321853    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g" event={"ID":"ad305e1a-a602-4c0c-9454-f0d783e281fb","Type":"ContainerStarted","Data":"981adbfd97d1c7b1f9ddfb0f60cda5d5971211e67d015157ad3b556c95dfbdc5"}
Oct 30 22:20:52.322079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.321909    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g" event={"ID":"ad305e1a-a602-4c0c-9454-f0d783e281fb","Type":"ContainerStarted","Data":"cb2767ba87cb908cb5cebc4baa33d5881616047be606c4cacf7d9d6f420c373b"}
Oct 30 22:20:52.322079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.321948    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"
Oct 30 22:20:52.409926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.409886    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"
Oct 30 22:20:52.439306 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.439253    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g" podStartSLOduration=1.439236562 podStartE2EDuration="1.439236562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:52.437579523 +0000 UTC m=+2303.171942556" watchObservedRunningTime="2026-10-30 22:20:52.439236562 +0000 UTC m=+2303.173599591"
Oct 30 22:20:52.796195 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:52.796168    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"]
Oct 30 22:20:52.797615 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:20:52.797587    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37cba11_3e73_47d7_88c8_133f37788561.slice/crio-8a4b1b20c8ff85212d3499c09027ccae1d45e3cb14a44a4bdf198f87e3a15fec WatchSource:0}: Error finding container 8a4b1b20c8ff85212d3499c09027ccae1d45e3cb14a44a4bdf198f87e3a15fec: Status 404 returned error can't find the container with id 8a4b1b20c8ff85212d3499c09027ccae1d45e3cb14a44a4bdf198f87e3a15fec
Oct 30 22:20:53.330430 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:53.330389    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh" event={"ID":"a37cba11-3e73-47d7-88c8-133f37788561","Type":"ContainerStarted","Data":"90ee3389baa6a3d8841e2a8390c88c88c1e07b2d515b2912b516dbd271ce7b37"}
Oct 30 22:20:53.330943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:53.330452    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh" event={"ID":"a37cba11-3e73-47d7-88c8-133f37788561","Type":"ContainerStarted","Data":"8a4b1b20c8ff85212d3499c09027ccae1d45e3cb14a44a4bdf198f87e3a15fec"}
Oct 30 22:20:53.330943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:53.330482    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"
Oct 30 22:20:53.451791 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:20:53.451734    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh" podStartSLOduration=1.451719143 podStartE2EDuration="1.451719143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:53.450071632 +0000 UTC m=+2304.184434662" watchObservedRunningTime="2026-10-30 22:20:53.451719143 +0000 UTC m=+2304.186082173"
Oct 30 22:20:58.202737 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:20:58.202693    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:21:02.519982 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:02.519950    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"]
Oct 30 22:21:02.520456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:02.520256    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk" podUID="756edb22-9e36-48e8-9835-a6570315b9cc" containerName="payload-processing" containerID="cri-o://e36763e1d4925b98d7926c5882c84ab334234c53291b6711c41946959dd58dcf" gracePeriod=30
Oct 30 22:21:02.571269 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:02.571234    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"]
Oct 30 22:21:02.571488 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:02.571467    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b" podUID="06b02b63-62f6-4575-be94-b976bad43919" containerName="payload-pre-processing" containerID="cri-o://cea4f74e365209b8294d66cb98663efb4f625c976205a44965d8bf3c5930c954" gracePeriod=30
Oct 30 22:21:02.911706 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:02.911660    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"
Oct 30 22:21:02.913532 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:02.913364    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/756edb22-9e36-48e8-9835-a6570315b9cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/756edb22-9e36-48e8-9835-a6570315b9cc-plugins-config-volume\") pod \"756edb22-9e36-48e8-9835-a6570315b9cc\" (UID: \"756edb22-9e36-48e8-9835-a6570315b9cc\") "
Oct 30 22:21:02.913532 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:02.913522    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756edb22-9e36-48e8-9835-a6570315b9cc-kube-api-access-hszfl\" (UniqueName: \"kubernetes.io/projected/756edb22-9e36-48e8-9835-a6570315b9cc-kube-api-access-hszfl\") pod \"756edb22-9e36-48e8-9835-a6570315b9cc\" (UID: \"756edb22-9e36-48e8-9835-a6570315b9cc\") "
Oct 30 22:21:02.913799 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:02.913774    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756edb22-9e36-48e8-9835-a6570315b9cc-plugins-config-volume" pod "756edb22-9e36-48e8-9835-a6570315b9cc" (UID: "756edb22-9e36-48e8-9835-a6570315b9cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:02.914236 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:02.914213    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/756edb22-9e36-48e8-9835-a6570315b9cc-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:02.915883 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:02.915861    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756edb22-9e36-48e8-9835-a6570315b9cc-kube-api-access-hszfl" pod "756edb22-9e36-48e8-9835-a6570315b9cc" (UID: "756edb22-9e36-48e8-9835-a6570315b9cc"). InnerVolumeSpecName "kube-api-access-hszfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:02.915974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:02.915865    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"
Oct 30 22:21:03.015096 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.015065    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06b02b63-62f6-4575-be94-b976bad43919-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b02b63-62f6-4575-be94-b976bad43919-plugins-config-volume\") pod \"06b02b63-62f6-4575-be94-b976bad43919\" (UID: \"06b02b63-62f6-4575-be94-b976bad43919\") "
Oct 30 22:21:03.015310 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.015111    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b02b63-62f6-4575-be94-b976bad43919-kube-api-access-sxnd8\" (UniqueName: \"kubernetes.io/projected/06b02b63-62f6-4575-be94-b976bad43919-kube-api-access-sxnd8\") pod \"06b02b63-62f6-4575-be94-b976bad43919\" (UID: \"06b02b63-62f6-4575-be94-b976bad43919\") "
Oct 30 22:21:03.015431 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.015407    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b02b63-62f6-4575-be94-b976bad43919-plugins-config-volume" pod "06b02b63-62f6-4575-be94-b976bad43919" (UID: "06b02b63-62f6-4575-be94-b976bad43919"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:03.015518 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.015497    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06b02b63-62f6-4575-be94-b976bad43919-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:03.015518 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.015520    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hszfl\" (UniqueName: \"kubernetes.io/projected/756edb22-9e36-48e8-9835-a6570315b9cc-kube-api-access-hszfl\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:03.017434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.017411    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b02b63-62f6-4575-be94-b976bad43919-kube-api-access-sxnd8" pod "06b02b63-62f6-4575-be94-b976bad43919" (UID: "06b02b63-62f6-4575-be94-b976bad43919"). InnerVolumeSpecName "kube-api-access-sxnd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:03.150160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.149317    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxnd8\" (UniqueName: \"kubernetes.io/projected/06b02b63-62f6-4575-be94-b976bad43919-kube-api-access-sxnd8\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:03.332845 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.332812    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"
Oct 30 22:21:03.401020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.400918    4553 generic.go:356] "Generic (PLEG): container finished" podID="756edb22-9e36-48e8-9835-a6570315b9cc" containerID="e36763e1d4925b98d7926c5882c84ab334234c53291b6711c41946959dd58dcf" exitCode=0
Oct 30 22:21:03.401020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.400971    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"
Oct 30 22:21:03.401020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.400992    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk" event={"ID":"756edb22-9e36-48e8-9835-a6570315b9cc","Type":"ContainerDied","Data":"e36763e1d4925b98d7926c5882c84ab334234c53291b6711c41946959dd58dcf"}
Oct 30 22:21:03.401367 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.401050    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk" event={"ID":"756edb22-9e36-48e8-9835-a6570315b9cc","Type":"ContainerDied","Data":"567e15a1ed619bb133fa10f5367cc55ea530aa321afffa776fb8b2ec4895bbb1"}
Oct 30 22:21:03.401367 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.401082    4553 scope.go:122] "RemoveContainer" containerID="e36763e1d4925b98d7926c5882c84ab334234c53291b6711c41946959dd58dcf"
Oct 30 22:21:03.402821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.402709    4553 generic.go:356] "Generic (PLEG): container finished" podID="06b02b63-62f6-4575-be94-b976bad43919" containerID="cea4f74e365209b8294d66cb98663efb4f625c976205a44965d8bf3c5930c954" exitCode=0
Oct 30 22:21:03.402821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.402741    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b" event={"ID":"06b02b63-62f6-4575-be94-b976bad43919","Type":"ContainerDied","Data":"cea4f74e365209b8294d66cb98663efb4f625c976205a44965d8bf3c5930c954"}
Oct 30 22:21:03.402821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.402772    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"
Oct 30 22:21:03.402821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.402788    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b" event={"ID":"06b02b63-62f6-4575-be94-b976bad43919","Type":"ContainerDied","Data":"46a5e04fd7f999f34cb197aab2cc823b048630f04422f64904489c09a88ea536"}
Oct 30 22:21:03.415673 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.415651    4553 scope.go:122] "RemoveContainer" containerID="e36763e1d4925b98d7926c5882c84ab334234c53291b6711c41946959dd58dcf"
Oct 30 22:21:03.416035 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:21:03.416011    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36763e1d4925b98d7926c5882c84ab334234c53291b6711c41946959dd58dcf\": container with ID starting with e36763e1d4925b98d7926c5882c84ab334234c53291b6711c41946959dd58dcf not found: ID does not exist" containerID="e36763e1d4925b98d7926c5882c84ab334234c53291b6711c41946959dd58dcf"
Oct 30 22:21:03.416098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.416046    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36763e1d4925b98d7926c5882c84ab334234c53291b6711c41946959dd58dcf"} err="failed to get container status \"e36763e1d4925b98d7926c5882c84ab334234c53291b6711c41946959dd58dcf\": rpc error: code = NotFound desc = could not find container \"e36763e1d4925b98d7926c5882c84ab334234c53291b6711c41946959dd58dcf\": container with ID starting with e36763e1d4925b98d7926c5882c84ab334234c53291b6711c41946959dd58dcf not found: ID does not exist"
Oct 30 22:21:03.416098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.416067    4553 scope.go:122] "RemoveContainer" containerID="cea4f74e365209b8294d66cb98663efb4f625c976205a44965d8bf3c5930c954"
Oct 30 22:21:03.427929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.427908    4553 scope.go:122] "RemoveContainer" containerID="cea4f74e365209b8294d66cb98663efb4f625c976205a44965d8bf3c5930c954"
Oct 30 22:21:03.428317 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:21:03.428290    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea4f74e365209b8294d66cb98663efb4f625c976205a44965d8bf3c5930c954\": container with ID starting with cea4f74e365209b8294d66cb98663efb4f625c976205a44965d8bf3c5930c954 not found: ID does not exist" containerID="cea4f74e365209b8294d66cb98663efb4f625c976205a44965d8bf3c5930c954"
Oct 30 22:21:03.428424 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.428323    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea4f74e365209b8294d66cb98663efb4f625c976205a44965d8bf3c5930c954"} err="failed to get container status \"cea4f74e365209b8294d66cb98663efb4f625c976205a44965d8bf3c5930c954\": rpc error: code = NotFound desc = could not find container \"cea4f74e365209b8294d66cb98663efb4f625c976205a44965d8bf3c5930c954\": container with ID starting with cea4f74e365209b8294d66cb98663efb4f625c976205a44965d8bf3c5930c954 not found: ID does not exist"
Oct 30 22:21:03.566367 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.566328    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"]
Oct 30 22:21:03.569367 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.569343    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-577ea8-6969d4d4cf-kqr2b"]
Oct 30 22:21:03.678897 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.678815    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"]
Oct 30 22:21:03.681667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:03.681646    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-577ea8-64c56bb4db-gf6tk"]
Oct 30 22:21:04.341858 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:04.341826    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"
Oct 30 22:21:05.394964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:05.394933    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06b02b63-62f6-4575-be94-b976bad43919" path="/var/lib/kubelet/pods/06b02b63-62f6-4575-be94-b976bad43919/volumes"
Oct 30 22:21:05.395433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:21:05.395417    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756edb22-9e36-48e8-9835-a6570315b9cc" path="/var/lib/kubelet/pods/756edb22-9e36-48e8-9835-a6570315b9cc/volumes"
Oct 30 22:21:28.203314 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:21:28.203275    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:21:58.203318 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:21:58.203278    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:22:16.717969 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:16.717868    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj_94e79eee-aaaf-49d0-b354-748bdfbe636e/payload-processing/0.log"
Oct 30 22:22:17.020433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:17.020357    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-hg9wk_05d7db25-7800-4217-a2fb-4396ba266c73/payload-processing/0.log"
Oct 30 22:22:17.278416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:17.278344    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-hg9wk_05d7db25-7800-4217-a2fb-4396ba266c73/payload-processing/0.log"
Oct 30 22:22:17.537362 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:17.537281    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh_a37cba11-3e73-47d7-88c8-133f37788561/payload-processing/0.log"
Oct 30 22:22:17.761433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:17.761402    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh_a37cba11-3e73-47d7-88c8-133f37788561/payload-processing/0.log"
Oct 30 22:22:23.941351 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:23.941303    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"]
Oct 30 22:22:23.941964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:23.941948    4553 cpu_manager.go:414] "containerMap: removing container" podUID="756edb22-9e36-48e8-9835-a6570315b9cc" containerName="payload-processing"
Oct 30 22:22:23.942008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:23.941964    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756edb22-9e36-48e8-9835-a6570315b9cc" containerName="payload-processing"
Oct 30 22:22:23.942008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:23.941994    4553 cpu_manager.go:414] "containerMap: removing container" podUID="06b02b63-62f6-4575-be94-b976bad43919" containerName="payload-pre-processing"
Oct 30 22:22:23.942008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:23.941999    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b02b63-62f6-4575-be94-b976bad43919" containerName="payload-pre-processing"
Oct 30 22:22:23.942123 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:23.942112    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="756edb22-9e36-48e8-9835-a6570315b9cc" containerName="payload-processing"
Oct 30 22:22:23.942184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:23.942128    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b02b63-62f6-4575-be94-b976bad43919" containerName="payload-pre-processing"
Oct 30 22:22:23.945840 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:23.945825    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"
Oct 30 22:22:24.025506 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.025461    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m8dw\" (UniqueName: \"kubernetes.io/projected/030bf2d1-9267-4507-86fa-c171a880b693-kube-api-access-4m8dw\") pod \"payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx\" (UID: \"030bf2d1-9267-4507-86fa-c171a880b693\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"
Oct 30 22:22:24.025691 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.025619    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030bf2d1-9267-4507-86fa-c171a880b693-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx\" (UID: \"030bf2d1-9267-4507-86fa-c171a880b693\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"
Oct 30 22:22:24.049566 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.049537    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8b1e0a\""
Oct 30 22:22:24.049566 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.049542    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8b1e0a-dockercfg-5wxzw\""
Oct 30 22:22:24.052710 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.052688    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"]
Oct 30 22:22:24.127294 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.127256    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030bf2d1-9267-4507-86fa-c171a880b693-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx\" (UID: \"030bf2d1-9267-4507-86fa-c171a880b693\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"
Oct 30 22:22:24.127480 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.127414    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m8dw\" (UniqueName: \"kubernetes.io/projected/030bf2d1-9267-4507-86fa-c171a880b693-kube-api-access-4m8dw\") pod \"payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx\" (UID: \"030bf2d1-9267-4507-86fa-c171a880b693\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"
Oct 30 22:22:24.128083 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.128064    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030bf2d1-9267-4507-86fa-c171a880b693-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx\" (UID: \"030bf2d1-9267-4507-86fa-c171a880b693\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"
Oct 30 22:22:24.189710 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.189671    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m8dw\" (UniqueName: \"kubernetes.io/projected/030bf2d1-9267-4507-86fa-c171a880b693-kube-api-access-4m8dw\") pod \"payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx\" (UID: \"030bf2d1-9267-4507-86fa-c171a880b693\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"
Oct 30 22:22:24.260512 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.260419    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"
Oct 30 22:22:24.643432 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.643406    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"]
Oct 30 22:22:24.645182 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:22:24.645149    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030bf2d1_9267_4507_86fa_c171a880b693.slice/crio-9622ecbde8485b963ebe36b336b675bb6e4ce8b76b104c5a09573477e4ca61b9 WatchSource:0}: Error finding container 9622ecbde8485b963ebe36b336b675bb6e4ce8b76b104c5a09573477e4ca61b9: Status 404 returned error can't find the container with id 9622ecbde8485b963ebe36b336b675bb6e4ce8b76b104c5a09573477e4ca61b9
Oct 30 22:22:24.691622 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.691591    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"]
Oct 30 22:22:24.697324 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.697302    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"
Oct 30 22:22:24.736528 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.736490    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268ee85a-c628-46c8-bae8-b92ed0456bb8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb\" (UID: \"268ee85a-c628-46c8-bae8-b92ed0456bb8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"
Oct 30 22:22:24.736662 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.736628    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54sj5\" (UniqueName: \"kubernetes.io/projected/268ee85a-c628-46c8-bae8-b92ed0456bb8-kube-api-access-54sj5\") pod \"payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb\" (UID: \"268ee85a-c628-46c8-bae8-b92ed0456bb8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"
Oct 30 22:22:24.759592 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.759561    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"]
Oct 30 22:22:24.837912 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.837868    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268ee85a-c628-46c8-bae8-b92ed0456bb8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb\" (UID: \"268ee85a-c628-46c8-bae8-b92ed0456bb8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"
Oct 30 22:22:24.838120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.837960    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54sj5\" (UniqueName: \"kubernetes.io/projected/268ee85a-c628-46c8-bae8-b92ed0456bb8-kube-api-access-54sj5\") pod \"payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb\" (UID: \"268ee85a-c628-46c8-bae8-b92ed0456bb8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"
Oct 30 22:22:24.838621 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.838596    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268ee85a-c628-46c8-bae8-b92ed0456bb8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb\" (UID: \"268ee85a-c628-46c8-bae8-b92ed0456bb8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"
Oct 30 22:22:24.899674 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.899610    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54sj5\" (UniqueName: \"kubernetes.io/projected/268ee85a-c628-46c8-bae8-b92ed0456bb8-kube-api-access-54sj5\") pod \"payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb\" (UID: \"268ee85a-c628-46c8-bae8-b92ed0456bb8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"
Oct 30 22:22:24.977799 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.977757    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx" event={"ID":"030bf2d1-9267-4507-86fa-c171a880b693","Type":"ContainerStarted","Data":"73b8c8979b3cc03e6a8d7b9c1f8ebae28dac5f9dc8991c6915ac2625eed84f0e"}
Oct 30 22:22:24.978344 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.977817    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx" event={"ID":"030bf2d1-9267-4507-86fa-c171a880b693","Type":"ContainerStarted","Data":"9622ecbde8485b963ebe36b336b675bb6e4ce8b76b104c5a09573477e4ca61b9"}
Oct 30 22:22:24.978344 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:24.977861    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"
Oct 30 22:22:25.029637 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:25.029603    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"
Oct 30 22:22:25.106803 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:25.106738    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx" podStartSLOduration=2.106713899 podStartE2EDuration="2.106713899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:25.101797682 +0000 UTC m=+2395.836160714" watchObservedRunningTime="2026-10-30 22:22:25.106713899 +0000 UTC m=+2395.841076932"
Oct 30 22:22:25.461333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:25.445036    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"]
Oct 30 22:22:25.461333 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:22:25.453987    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268ee85a_c628_46c8_bae8_b92ed0456bb8.slice/crio-75aeda6a0800274911e593dd917c7cb030c58275501ac40a618ba8218ff3b4ff WatchSource:0}: Error finding container 75aeda6a0800274911e593dd917c7cb030c58275501ac40a618ba8218ff3b4ff: Status 404 returned error can't find the container with id 75aeda6a0800274911e593dd917c7cb030c58275501ac40a618ba8218ff3b4ff
Oct 30 22:22:25.636161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:25.631821    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"]
Oct 30 22:22:25.636161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:25.632048    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp" podUID="5b89e92c-e8c0-4282-b6e5-a4c8f867e08b" containerName="payload-processing" containerID="cri-o://e98a167502aebf771a562f0ca0635da95b7b854718dd292f9f86ac56f989ce9c" gracePeriod=30
Oct 30 22:22:25.698652 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:25.698581    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"]
Oct 30 22:22:25.698811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:25.698765    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k" podUID="1dd04861-5971-4a70-9a64-ce27e2f9f6b2" containerName="payload-pre-processing" containerID="cri-o://b437a265d2aa5c89b69970b26f7685a9050fbb58aa163ccb4885c1911b5b09ad" gracePeriod=30
Oct 30 22:22:25.995110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:25.992361    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb" event={"ID":"268ee85a-c628-46c8-bae8-b92ed0456bb8","Type":"ContainerStarted","Data":"e67352530a43d3f62413e851a4c37e84f66934c3763c2372d24f4ea1169ef22c"}
Oct 30 22:22:25.995110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:25.992440    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb" event={"ID":"268ee85a-c628-46c8-bae8-b92ed0456bb8","Type":"ContainerStarted","Data":"75aeda6a0800274911e593dd917c7cb030c58275501ac40a618ba8218ff3b4ff"}
Oct 30 22:22:25.995110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:25.993452    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"
Oct 30 22:22:25.995110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:25.994575    4553 generic.go:356] "Generic (PLEG): container finished" podID="1dd04861-5971-4a70-9a64-ce27e2f9f6b2" containerID="b437a265d2aa5c89b69970b26f7685a9050fbb58aa163ccb4885c1911b5b09ad" exitCode=0
Oct 30 22:22:25.995110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:25.994684    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k" event={"ID":"1dd04861-5971-4a70-9a64-ce27e2f9f6b2","Type":"ContainerDied","Data":"b437a265d2aa5c89b69970b26f7685a9050fbb58aa163ccb4885c1911b5b09ad"}
Oct 30 22:22:25.996325 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:25.996305    4553 generic.go:356] "Generic (PLEG): container finished" podID="5b89e92c-e8c0-4282-b6e5-a4c8f867e08b" containerID="e98a167502aebf771a562f0ca0635da95b7b854718dd292f9f86ac56f989ce9c" exitCode=0
Oct 30 22:22:25.996438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:25.996381    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp" event={"ID":"5b89e92c-e8c0-4282-b6e5-a4c8f867e08b","Type":"ContainerDied","Data":"e98a167502aebf771a562f0ca0635da95b7b854718dd292f9f86ac56f989ce9c"}
Oct 30 22:22:26.029377 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:26.029350    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"
Oct 30 22:22:26.071390 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:26.071363    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"
Oct 30 22:22:26.116287 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:26.116234    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb" podStartSLOduration=2.116216866 podStartE2EDuration="2.116216866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:26.113894976 +0000 UTC m=+2396.848258007" watchObservedRunningTime="2026-10-30 22:22:26.116216866 +0000 UTC m=+2396.850579899"
Oct 30 22:22:26.209447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:26.209415    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b89e92c-e8c0-4282-b6e5-a4c8f867e08b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b89e92c-e8c0-4282-b6e5-a4c8f867e08b-plugins-config-volume\") pod \"5b89e92c-e8c0-4282-b6e5-a4c8f867e08b\" (UID: \"5b89e92c-e8c0-4282-b6e5-a4c8f867e08b\") "
Oct 30 22:22:26.209619 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:26.209524    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b89e92c-e8c0-4282-b6e5-a4c8f867e08b-kube-api-access-47b5g\" (UniqueName: \"kubernetes.io/projected/5b89e92c-e8c0-4282-b6e5-a4c8f867e08b-kube-api-access-47b5g\") pod \"5b89e92c-e8c0-4282-b6e5-a4c8f867e08b\" (UID: \"5b89e92c-e8c0-4282-b6e5-a4c8f867e08b\") "
Oct 30 22:22:26.209619 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:26.209570    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dd04861-5971-4a70-9a64-ce27e2f9f6b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd04861-5971-4a70-9a64-ce27e2f9f6b2-plugins-config-volume\") pod \"1dd04861-5971-4a70-9a64-ce27e2f9f6b2\" (UID: \"1dd04861-5971-4a70-9a64-ce27e2f9f6b2\") "
Oct 30 22:22:26.209738 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:26.209640    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dd04861-5971-4a70-9a64-ce27e2f9f6b2-kube-api-access-pp5tl\" (UniqueName: \"kubernetes.io/projected/1dd04861-5971-4a70-9a64-ce27e2f9f6b2-kube-api-access-pp5tl\") pod \"1dd04861-5971-4a70-9a64-ce27e2f9f6b2\" (UID: \"1dd04861-5971-4a70-9a64-ce27e2f9f6b2\") "
Oct 30 22:22:26.209853 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:26.209826    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b89e92c-e8c0-4282-b6e5-a4c8f867e08b-plugins-config-volume" pod "5b89e92c-e8c0-4282-b6e5-a4c8f867e08b" (UID: "5b89e92c-e8c0-4282-b6e5-a4c8f867e08b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:26.210034 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:26.210002    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd04861-5971-4a70-9a64-ce27e2f9f6b2-plugins-config-volume" pod "1dd04861-5971-4a70-9a64-ce27e2f9f6b2" (UID: "1dd04861-5971-4a70-9a64-ce27e2f9f6b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:26.210216 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:26.210194    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b89e92c-e8c0-4282-b6e5-a4c8f867e08b-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:26.210368 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:26.210218    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd04861-5971-4a70-9a64-ce27e2f9f6b2-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:26.212535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:26.212512    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b89e92c-e8c0-4282-b6e5-a4c8f867e08b-kube-api-access-47b5g" pod "5b89e92c-e8c0-4282-b6e5-a4c8f867e08b" (UID: "5b89e92c-e8c0-4282-b6e5-a4c8f867e08b"). InnerVolumeSpecName "kube-api-access-47b5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:26.212654 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:26.212534    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd04861-5971-4a70-9a64-ce27e2f9f6b2-kube-api-access-pp5tl" pod "1dd04861-5971-4a70-9a64-ce27e2f9f6b2" (UID: "1dd04861-5971-4a70-9a64-ce27e2f9f6b2"). InnerVolumeSpecName "kube-api-access-pp5tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:26.311531 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:26.311436    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp5tl\" (UniqueName: \"kubernetes.io/projected/1dd04861-5971-4a70-9a64-ce27e2f9f6b2-kube-api-access-pp5tl\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:26.311531 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:26.311474    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47b5g\" (UniqueName: \"kubernetes.io/projected/5b89e92c-e8c0-4282-b6e5-a4c8f867e08b-kube-api-access-47b5g\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:27.004440 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:27.004409    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"
Oct 30 22:22:27.004962 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:27.004406    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k" event={"ID":"1dd04861-5971-4a70-9a64-ce27e2f9f6b2","Type":"ContainerDied","Data":"94d8502d12a35ddf84b822826d9a1d495c0d2f263b075bea261dbd1e234bc746"}
Oct 30 22:22:27.004962 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:27.004580    4553 scope.go:122] "RemoveContainer" containerID="b437a265d2aa5c89b69970b26f7685a9050fbb58aa163ccb4885c1911b5b09ad"
Oct 30 22:22:27.006268 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:27.006242    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp" event={"ID":"5b89e92c-e8c0-4282-b6e5-a4c8f867e08b","Type":"ContainerDied","Data":"6e05abf42524b7274826d847fcd81ebb0bc703eb447f0392e6660d591f732bbf"}
Oct 30 22:22:27.006391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:27.006311    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"
Oct 30 22:22:27.016858 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:27.016835    4553 scope.go:122] "RemoveContainer" containerID="e98a167502aebf771a562f0ca0635da95b7b854718dd292f9f86ac56f989ce9c"
Oct 30 22:22:27.171777 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:27.171741    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"]
Oct 30 22:22:27.174708 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:27.174679    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69a1d2-fb9b8d85b-2tszp"]
Oct 30 22:22:27.285486 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:27.285387    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"]
Oct 30 22:22:27.290636 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:27.290581    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-69a1d2-594bb58c47-wvs7k"]
Oct 30 22:22:27.395357 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:27.395325    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dd04861-5971-4a70-9a64-ce27e2f9f6b2" path="/var/lib/kubelet/pods/1dd04861-5971-4a70-9a64-ce27e2f9f6b2/volumes"
Oct 30 22:22:27.395866 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:27.395847    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b89e92c-e8c0-4282-b6e5-a4c8f867e08b" path="/var/lib/kubelet/pods/5b89e92c-e8c0-4282-b6e5-a4c8f867e08b/volumes"
Oct 30 22:22:28.202574 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:22:28.202541    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:22:29.794899 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:29.794870    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:22:29.800246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:29.800221    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:22:35.999420 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:35.999386    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"
Oct 30 22:22:37.023317 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:37.018952    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"
Oct 30 22:22:57.139069 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.139028    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"]
Oct 30 22:22:57.139652 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.139361    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7" podUID="621f7d89-0683-4902-8d4d-078aa96761f4" containerName="maas-api" containerID="cri-o://fa87c17796bf4e6acacf386f0131e3a719fb0dc2ed847e578479afe65dc92480" gracePeriod=30
Oct 30 22:22:57.461654 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.461622    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:22:57.610829 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.610804    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621f7d89-0683-4902-8d4d-078aa96761f4-kube-api-access-82h7c\" (UniqueName: \"kubernetes.io/projected/621f7d89-0683-4902-8d4d-078aa96761f4-kube-api-access-82h7c\") pod \"621f7d89-0683-4902-8d4d-078aa96761f4\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") "
Oct 30 22:22:57.611011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.610840    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-maas-api-tls\") pod \"621f7d89-0683-4902-8d4d-078aa96761f4\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") "
Oct 30 22:22:57.611011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.610887    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621f7d89-0683-4902-8d4d-078aa96761f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621f7d89-0683-4902-8d4d-078aa96761f4-openshift-service-ca\") pod \"621f7d89-0683-4902-8d4d-078aa96761f4\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") "
Oct 30 22:22:57.611011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.610950    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-metrics-certs\") pod \"621f7d89-0683-4902-8d4d-078aa96761f4\" (UID: \"621f7d89-0683-4902-8d4d-078aa96761f4\") "
Oct 30 22:22:57.611637 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.611607    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621f7d89-0683-4902-8d4d-078aa96761f4-openshift-service-ca" pod "621f7d89-0683-4902-8d4d-078aa96761f4" (UID: "621f7d89-0683-4902-8d4d-078aa96761f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:57.613614 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.613590    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"]
Oct 30 22:22:57.613901 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.613844    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg" podUID="09d3a76e-ef0c-4ec6-b8c4-322e180ad95a" containerName="payload-processing" containerID="cri-o://30fe2729a68c9e7a83cda2c11404830c7eb612fa6089634b58aa8dc5496b3b1d" gracePeriod=30
Oct 30 22:22:57.619536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.619502    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-maas-api-tls" pod "621f7d89-0683-4902-8d4d-078aa96761f4" (UID: "621f7d89-0683-4902-8d4d-078aa96761f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:57.619648 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.619539    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621f7d89-0683-4902-8d4d-078aa96761f4-kube-api-access-82h7c" pod "621f7d89-0683-4902-8d4d-078aa96761f4" (UID: "621f7d89-0683-4902-8d4d-078aa96761f4"). InnerVolumeSpecName "kube-api-access-82h7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:57.621568 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.621538    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-metrics-certs" pod "621f7d89-0683-4902-8d4d-078aa96761f4" (UID: "621f7d89-0683-4902-8d4d-078aa96761f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:57.671023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.670989    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"]
Oct 30 22:22:57.671213 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.671194    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh" podUID="dc594a39-569b-4782-8396-4074ac150c4a" containerName="payload-pre-processing" containerID="cri-o://5991767be767c2ff85cd218c2fd40faca559cf93f16cb1069182fd7ba99017dc" gracePeriod=30
Oct 30 22:22:57.713120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.713080    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/621f7d89-0683-4902-8d4d-078aa96761f4-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:57.713120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.713124    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:57.713120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.713158    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82h7c\" (UniqueName: \"kubernetes.io/projected/621f7d89-0683-4902-8d4d-078aa96761f4-kube-api-access-82h7c\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:57.713120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.713171    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/621f7d89-0683-4902-8d4d-078aa96761f4-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:57.931947 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.931921    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"
Oct 30 22:22:57.971465 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:57.971438    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"
Oct 30 22:22:58.018273 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.018240    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d3a76e-ef0c-4ec6-b8c4-322e180ad95a-kube-api-access-hv9j8\" (UniqueName: \"kubernetes.io/projected/09d3a76e-ef0c-4ec6-b8c4-322e180ad95a-kube-api-access-hv9j8\") pod \"09d3a76e-ef0c-4ec6-b8c4-322e180ad95a\" (UID: \"09d3a76e-ef0c-4ec6-b8c4-322e180ad95a\") "
Oct 30 22:22:58.018463 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.018429    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d3a76e-ef0c-4ec6-b8c4-322e180ad95a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d3a76e-ef0c-4ec6-b8c4-322e180ad95a-plugins-config-volume\") pod \"09d3a76e-ef0c-4ec6-b8c4-322e180ad95a\" (UID: \"09d3a76e-ef0c-4ec6-b8c4-322e180ad95a\") "
Oct 30 22:22:58.018822 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.018798    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d3a76e-ef0c-4ec6-b8c4-322e180ad95a-plugins-config-volume" pod "09d3a76e-ef0c-4ec6-b8c4-322e180ad95a" (UID: "09d3a76e-ef0c-4ec6-b8c4-322e180ad95a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:58.018935 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.018923    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d3a76e-ef0c-4ec6-b8c4-322e180ad95a-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:58.020532 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.020513    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d3a76e-ef0c-4ec6-b8c4-322e180ad95a-kube-api-access-hv9j8" pod "09d3a76e-ef0c-4ec6-b8c4-322e180ad95a" (UID: "09d3a76e-ef0c-4ec6-b8c4-322e180ad95a"). InnerVolumeSpecName "kube-api-access-hv9j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:58.119572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.119539    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc594a39-569b-4782-8396-4074ac150c4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc594a39-569b-4782-8396-4074ac150c4a-plugins-config-volume\") pod \"dc594a39-569b-4782-8396-4074ac150c4a\" (UID: \"dc594a39-569b-4782-8396-4074ac150c4a\") "
Oct 30 22:22:58.119744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.119657    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc594a39-569b-4782-8396-4074ac150c4a-kube-api-access-xmhrp\" (UniqueName: \"kubernetes.io/projected/dc594a39-569b-4782-8396-4074ac150c4a-kube-api-access-xmhrp\") pod \"dc594a39-569b-4782-8396-4074ac150c4a\" (UID: \"dc594a39-569b-4782-8396-4074ac150c4a\") "
Oct 30 22:22:58.120029 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.119986    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc594a39-569b-4782-8396-4074ac150c4a-plugins-config-volume" pod "dc594a39-569b-4782-8396-4074ac150c4a" (UID: "dc594a39-569b-4782-8396-4074ac150c4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:58.120029 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.120000    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv9j8\" (UniqueName: \"kubernetes.io/projected/09d3a76e-ef0c-4ec6-b8c4-322e180ad95a-kube-api-access-hv9j8\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:58.121995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.121975    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc594a39-569b-4782-8396-4074ac150c4a-kube-api-access-xmhrp" pod "dc594a39-569b-4782-8396-4074ac150c4a" (UID: "dc594a39-569b-4782-8396-4074ac150c4a"). InnerVolumeSpecName "kube-api-access-xmhrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:58.208259 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:22:58.208218    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:22:58.221297 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.221224    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc594a39-569b-4782-8396-4074ac150c4a-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:58.221297 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.221249    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmhrp\" (UniqueName: \"kubernetes.io/projected/dc594a39-569b-4782-8396-4074ac150c4a-kube-api-access-xmhrp\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:22:58.229553 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.229523    4553 generic.go:356] "Generic (PLEG): container finished" podID="621f7d89-0683-4902-8d4d-078aa96761f4" containerID="fa87c17796bf4e6acacf386f0131e3a719fb0dc2ed847e578479afe65dc92480" exitCode=0
Oct 30 22:22:58.229698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.229577    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"
Oct 30 22:22:58.229698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.229598    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7" event={"ID":"621f7d89-0683-4902-8d4d-078aa96761f4","Type":"ContainerDied","Data":"fa87c17796bf4e6acacf386f0131e3a719fb0dc2ed847e578479afe65dc92480"}
Oct 30 22:22:58.229698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.229652    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7" event={"ID":"621f7d89-0683-4902-8d4d-078aa96761f4","Type":"ContainerDied","Data":"2b183b059a6dcfbb27ce2b882ba82c3f00e46b8d8eed0c41313b2f86fc872a25"}
Oct 30 22:22:58.229698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.229673    4553 scope.go:122] "RemoveContainer" containerID="fa87c17796bf4e6acacf386f0131e3a719fb0dc2ed847e578479afe65dc92480"
Oct 30 22:22:58.231369 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.231334    4553 generic.go:356] "Generic (PLEG): container finished" podID="dc594a39-569b-4782-8396-4074ac150c4a" containerID="5991767be767c2ff85cd218c2fd40faca559cf93f16cb1069182fd7ba99017dc" exitCode=0
Oct 30 22:22:58.231499 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.231386    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"
Oct 30 22:22:58.231499 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.231388    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh" event={"ID":"dc594a39-569b-4782-8396-4074ac150c4a","Type":"ContainerDied","Data":"5991767be767c2ff85cd218c2fd40faca559cf93f16cb1069182fd7ba99017dc"}
Oct 30 22:22:58.231499 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.231460    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh" event={"ID":"dc594a39-569b-4782-8396-4074ac150c4a","Type":"ContainerDied","Data":"bb749cd91e80b7dc2fb77cf1c4de00f95a13d44ec75c343ffd9ee8f4aba6d422"}
Oct 30 22:22:58.233217 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.233115    4553 generic.go:356] "Generic (PLEG): container finished" podID="09d3a76e-ef0c-4ec6-b8c4-322e180ad95a" containerID="30fe2729a68c9e7a83cda2c11404830c7eb612fa6089634b58aa8dc5496b3b1d" exitCode=0
Oct 30 22:22:58.233335 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.233171    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"
Oct 30 22:22:58.233422 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.233167    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg" event={"ID":"09d3a76e-ef0c-4ec6-b8c4-322e180ad95a","Type":"ContainerDied","Data":"30fe2729a68c9e7a83cda2c11404830c7eb612fa6089634b58aa8dc5496b3b1d"}
Oct 30 22:22:58.233422 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.233386    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg" event={"ID":"09d3a76e-ef0c-4ec6-b8c4-322e180ad95a","Type":"ContainerDied","Data":"6203aa2f040fa638263402aeb0f5a30ef940585af8abbde2a6dc6e6720fb54f7"}
Oct 30 22:22:58.241378 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.241357    4553 scope.go:122] "RemoveContainer" containerID="fa87c17796bf4e6acacf386f0131e3a719fb0dc2ed847e578479afe65dc92480"
Oct 30 22:22:58.241717 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:22:58.241681    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa87c17796bf4e6acacf386f0131e3a719fb0dc2ed847e578479afe65dc92480\": container with ID starting with fa87c17796bf4e6acacf386f0131e3a719fb0dc2ed847e578479afe65dc92480 not found: ID does not exist" containerID="fa87c17796bf4e6acacf386f0131e3a719fb0dc2ed847e578479afe65dc92480"
Oct 30 22:22:58.241819 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.241727    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa87c17796bf4e6acacf386f0131e3a719fb0dc2ed847e578479afe65dc92480"} err="failed to get container status \"fa87c17796bf4e6acacf386f0131e3a719fb0dc2ed847e578479afe65dc92480\": rpc error: code = NotFound desc = could not find container \"fa87c17796bf4e6acacf386f0131e3a719fb0dc2ed847e578479afe65dc92480\": container with ID starting with fa87c17796bf4e6acacf386f0131e3a719fb0dc2ed847e578479afe65dc92480 not found: ID does not exist"
Oct 30 22:22:58.241819 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.241746    4553 scope.go:122] "RemoveContainer" containerID="5991767be767c2ff85cd218c2fd40faca559cf93f16cb1069182fd7ba99017dc"
Oct 30 22:22:58.255405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.255383    4553 scope.go:122] "RemoveContainer" containerID="5991767be767c2ff85cd218c2fd40faca559cf93f16cb1069182fd7ba99017dc"
Oct 30 22:22:58.255727 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:22:58.255700    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5991767be767c2ff85cd218c2fd40faca559cf93f16cb1069182fd7ba99017dc\": container with ID starting with 5991767be767c2ff85cd218c2fd40faca559cf93f16cb1069182fd7ba99017dc not found: ID does not exist" containerID="5991767be767c2ff85cd218c2fd40faca559cf93f16cb1069182fd7ba99017dc"
Oct 30 22:22:58.255825 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.255738    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5991767be767c2ff85cd218c2fd40faca559cf93f16cb1069182fd7ba99017dc"} err="failed to get container status \"5991767be767c2ff85cd218c2fd40faca559cf93f16cb1069182fd7ba99017dc\": rpc error: code = NotFound desc = could not find container \"5991767be767c2ff85cd218c2fd40faca559cf93f16cb1069182fd7ba99017dc\": container with ID starting with 5991767be767c2ff85cd218c2fd40faca559cf93f16cb1069182fd7ba99017dc not found: ID does not exist"
Oct 30 22:22:58.255825 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.255765    4553 scope.go:122] "RemoveContainer" containerID="30fe2729a68c9e7a83cda2c11404830c7eb612fa6089634b58aa8dc5496b3b1d"
Oct 30 22:22:58.272240 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.272014    4553 scope.go:122] "RemoveContainer" containerID="30fe2729a68c9e7a83cda2c11404830c7eb612fa6089634b58aa8dc5496b3b1d"
Oct 30 22:22:58.272826 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:22:58.272721    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30fe2729a68c9e7a83cda2c11404830c7eb612fa6089634b58aa8dc5496b3b1d\": container with ID starting with 30fe2729a68c9e7a83cda2c11404830c7eb612fa6089634b58aa8dc5496b3b1d not found: ID does not exist" containerID="30fe2729a68c9e7a83cda2c11404830c7eb612fa6089634b58aa8dc5496b3b1d"
Oct 30 22:22:58.272826 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.272787    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30fe2729a68c9e7a83cda2c11404830c7eb612fa6089634b58aa8dc5496b3b1d"} err="failed to get container status \"30fe2729a68c9e7a83cda2c11404830c7eb612fa6089634b58aa8dc5496b3b1d\": rpc error: code = NotFound desc = could not find container \"30fe2729a68c9e7a83cda2c11404830c7eb612fa6089634b58aa8dc5496b3b1d\": container with ID starting with 30fe2729a68c9e7a83cda2c11404830c7eb612fa6089634b58aa8dc5496b3b1d not found: ID does not exist"
Oct 30 22:22:58.416649 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.416620    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"]
Oct 30 22:22:58.419598 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.419553    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a28730-5945b75fff-gtdh7"]
Oct 30 22:22:58.531910 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.531820    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"]
Oct 30 22:22:58.536309 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.536280    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a28730-797866c57d-7vgzg"]
Oct 30 22:22:58.646269 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.646152    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"]
Oct 30 22:22:58.649555 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:58.649530    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a28730-64694c8c9f-69dbh"]
Oct 30 22:22:59.401889 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:59.401844    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d3a76e-ef0c-4ec6-b8c4-322e180ad95a" path="/var/lib/kubelet/pods/09d3a76e-ef0c-4ec6-b8c4-322e180ad95a/volumes"
Oct 30 22:22:59.402501 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:59.402450    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621f7d89-0683-4902-8d4d-078aa96761f4" path="/var/lib/kubelet/pods/621f7d89-0683-4902-8d4d-078aa96761f4/volumes"
Oct 30 22:22:59.402953 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:22:59.402934    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc594a39-569b-4782-8396-4074ac150c4a" path="/var/lib/kubelet/pods/dc594a39-569b-4782-8396-4074ac150c4a/volumes"
Oct 30 22:23:04.057802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.057761    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"]
Oct 30 22:23:04.058429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.058414    4553 cpu_manager.go:414] "containerMap: removing container" podUID="621f7d89-0683-4902-8d4d-078aa96761f4" containerName="maas-api"
Oct 30 22:23:04.058476 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.058430    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621f7d89-0683-4902-8d4d-078aa96761f4" containerName="maas-api"
Oct 30 22:23:04.058476 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.058441    4553 cpu_manager.go:414] "containerMap: removing container" podUID="1dd04861-5971-4a70-9a64-ce27e2f9f6b2" containerName="payload-pre-processing"
Oct 30 22:23:04.058476 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.058447    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dd04861-5971-4a70-9a64-ce27e2f9f6b2" containerName="payload-pre-processing"
Oct 30 22:23:04.058476 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.058460    4553 cpu_manager.go:414] "containerMap: removing container" podUID="5b89e92c-e8c0-4282-b6e5-a4c8f867e08b" containerName="payload-processing"
Oct 30 22:23:04.058476 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.058465    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b89e92c-e8c0-4282-b6e5-a4c8f867e08b" containerName="payload-processing"
Oct 30 22:23:04.058625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.058486    4553 cpu_manager.go:414] "containerMap: removing container" podUID="09d3a76e-ef0c-4ec6-b8c4-322e180ad95a" containerName="payload-processing"
Oct 30 22:23:04.058625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.058490    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d3a76e-ef0c-4ec6-b8c4-322e180ad95a" containerName="payload-processing"
Oct 30 22:23:04.058625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.058502    4553 cpu_manager.go:414] "containerMap: removing container" podUID="dc594a39-569b-4782-8396-4074ac150c4a" containerName="payload-pre-processing"
Oct 30 22:23:04.058625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.058507    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc594a39-569b-4782-8396-4074ac150c4a" containerName="payload-pre-processing"
Oct 30 22:23:04.058625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.058592    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b89e92c-e8c0-4282-b6e5-a4c8f867e08b" containerName="payload-processing"
Oct 30 22:23:04.058625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.058605    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d3a76e-ef0c-4ec6-b8c4-322e180ad95a" containerName="payload-processing"
Oct 30 22:23:04.058625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.058614    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc594a39-569b-4782-8396-4074ac150c4a" containerName="payload-pre-processing"
Oct 30 22:23:04.058625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.058624    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="621f7d89-0683-4902-8d4d-078aa96761f4" containerName="maas-api"
Oct 30 22:23:04.058884 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.058631    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dd04861-5971-4a70-9a64-ce27e2f9f6b2" containerName="payload-pre-processing"
Oct 30 22:23:04.063790 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.063773    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:04.106709 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.106679    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-086575-c466fbd4-wd76c\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:04.106863 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.106727    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-metrics-certs\") pod \"maas-api-e2e-praxis-a-086575-c466fbd4-wd76c\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:04.106921 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.106884    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpw6q\" (UniqueName: \"kubernetes.io/projected/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-kube-api-access-xpw6q\") pod \"maas-api-e2e-praxis-a-086575-c466fbd4-wd76c\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:04.106971 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.106950    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-086575-c466fbd4-wd76c\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:04.117703 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.117674    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-086575-serving-cert\""
Oct 30 22:23:04.121449 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.121425    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"]
Oct 30 22:23:04.208434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.208395    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-086575-c466fbd4-wd76c\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:04.208625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.208468    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-metrics-certs\") pod \"maas-api-e2e-praxis-a-086575-c466fbd4-wd76c\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:04.208625 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:23:04.208525    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-serving-cert: secret "maas-api-e2e-praxis-a-086575-serving-cert" not found
Oct 30 22:23:04.208625 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:23:04.208616    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-maas-api-tls podName:73f4ca98-5d19-4c61-a5bd-6421cd5198d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:23:04.708592508 +0000 UTC m=+2435.442955530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-maas-api-tls") pod "maas-api-e2e-praxis-a-086575-c466fbd4-wd76c" (UID: "73f4ca98-5d19-4c61-a5bd-6421cd5198d9") : secret "maas-api-e2e-praxis-a-086575-serving-cert" not found
Oct 30 22:23:04.208837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.208639    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpw6q\" (UniqueName: \"kubernetes.io/projected/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-kube-api-access-xpw6q\") pod \"maas-api-e2e-praxis-a-086575-c466fbd4-wd76c\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:04.208837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.208721    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-086575-c466fbd4-wd76c\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:04.221749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.209513    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-086575-c466fbd4-wd76c\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:04.221749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.212088    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-metrics-certs\") pod \"maas-api-e2e-praxis-a-086575-c466fbd4-wd76c\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:04.267884 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.267851    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpw6q\" (UniqueName: \"kubernetes.io/projected/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-kube-api-access-xpw6q\") pod \"maas-api-e2e-praxis-a-086575-c466fbd4-wd76c\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:04.722800 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.722760    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-086575-c466fbd4-wd76c\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:04.725553 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.725526    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-maas-api-tls\") pod \"maas-api-e2e-praxis-a-086575-c466fbd4-wd76c\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:04.977964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:04.977878    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:05.362529 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:05.362504    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"]
Oct 30 22:23:05.364767 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:23:05.364736    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f4ca98_5d19_4c61_a5bd_6421cd5198d9.slice/crio-281bca97efc3074c9792368d2f826d2bea2b8cc333d5aad3964ca1d1e9f88eb5 WatchSource:0}: Error finding container 281bca97efc3074c9792368d2f826d2bea2b8cc333d5aad3964ca1d1e9f88eb5: Status 404 returned error can't find the container with id 281bca97efc3074c9792368d2f826d2bea2b8cc333d5aad3964ca1d1e9f88eb5
Oct 30 22:23:05.366226 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:05.366209    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:23:06.293369 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:06.293331    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c" event={"ID":"73f4ca98-5d19-4c61-a5bd-6421cd5198d9","Type":"ContainerStarted","Data":"281bca97efc3074c9792368d2f826d2bea2b8cc333d5aad3964ca1d1e9f88eb5"}
Oct 30 22:23:07.296642 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:07.296605    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"]
Oct 30 22:23:07.297736 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:07.296895    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2" podUID="45c033dd-33b5-4002-8161-d8a9402c7cd4" containerName="istio-proxy" containerID="cri-o://17dad28906f86142c6d81d85e3b379df72eb47d2288303f28b55b40bd0ef9d92" gracePeriod=30
Oct 30 22:23:07.303059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:07.303032    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c" event={"ID":"73f4ca98-5d19-4c61-a5bd-6421cd5198d9","Type":"ContainerStarted","Data":"394fc1eab9c846d098b36d135af03b2a5b5a610ff1b6a07e70c91b96804e924d"}
Oct 30 22:23:07.303198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:07.303083    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:07.547843 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:07.547733    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c" podStartSLOduration=2.599587562 podStartE2EDuration="3.547716087s" totalImagesPullingTime="948.128525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:07.546312994 +0000 UTC m=+2438.280676025" watchObservedRunningTime="2026-10-30 22:23:07.547716087 +0000 UTC m=+2438.282079148"
Oct 30 22:23:12.564778 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.564713    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:23:12.679221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.679187    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45c033dd-33b5-4002-8161-d8a9402c7cd4-kube-api-access-576nm\" (UniqueName: \"kubernetes.io/projected/45c033dd-33b5-4002-8161-d8a9402c7cd4-kube-api-access-576nm\") pod \"45c033dd-33b5-4002-8161-d8a9402c7cd4\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") "
Oct 30 22:23:12.679436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.679251    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-podinfo\") pod \"45c033dd-33b5-4002-8161-d8a9402c7cd4\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") "
Oct 30 22:23:12.679436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.679286    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-workload-socket\") pod \"45c033dd-33b5-4002-8161-d8a9402c7cd4\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") "
Oct 30 22:23:12.679436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.679364    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-token\" (UniqueName: \"kubernetes.io/projected/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-token\") pod \"45c033dd-33b5-4002-8161-d8a9402c7cd4\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") "
Oct 30 22:23:12.679620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.679451    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-data\") pod \"45c033dd-33b5-4002-8161-d8a9402c7cd4\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") "
Oct 30 22:23:12.679620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.679475    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-workload-certs\") pod \"45c033dd-33b5-4002-8161-d8a9402c7cd4\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") "
Oct 30 22:23:12.679620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.679522    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-credential-socket\") pod \"45c033dd-33b5-4002-8161-d8a9402c7cd4\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") "
Oct 30 22:23:12.679620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.679613    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-envoy\") pod \"45c033dd-33b5-4002-8161-d8a9402c7cd4\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") "
Oct 30 22:23:12.679830 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.679617    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-workload-socket" pod "45c033dd-33b5-4002-8161-d8a9402c7cd4" (UID: "45c033dd-33b5-4002-8161-d8a9402c7cd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:12.679830 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.679645    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c033dd-33b5-4002-8161-d8a9402c7cd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c033dd-33b5-4002-8161-d8a9402c7cd4-istiod-ca-cert\") pod \"45c033dd-33b5-4002-8161-d8a9402c7cd4\" (UID: \"45c033dd-33b5-4002-8161-d8a9402c7cd4\") "
Oct 30 22:23:12.679830 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.679724    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-data" pod "45c033dd-33b5-4002-8161-d8a9402c7cd4" (UID: "45c033dd-33b5-4002-8161-d8a9402c7cd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:12.680174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.679794    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-workload-certs" pod "45c033dd-33b5-4002-8161-d8a9402c7cd4" (UID: "45c033dd-33b5-4002-8161-d8a9402c7cd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:12.680174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.679870    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-credential-socket" pod "45c033dd-33b5-4002-8161-d8a9402c7cd4" (UID: "45c033dd-33b5-4002-8161-d8a9402c7cd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:12.680174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.680154    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c033dd-33b5-4002-8161-d8a9402c7cd4-istiod-ca-cert" pod "45c033dd-33b5-4002-8161-d8a9402c7cd4" (UID: "45c033dd-33b5-4002-8161-d8a9402c7cd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:12.680486 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.680461    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45c033dd-33b5-4002-8161-d8a9402c7cd4-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:12.680559 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.680493    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:12.680559 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.680509    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:12.680559 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.680526    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:12.680559 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.680541    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:12.681682 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.681652    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-podinfo" pod "45c033dd-33b5-4002-8161-d8a9402c7cd4" (UID: "45c033dd-33b5-4002-8161-d8a9402c7cd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:23:12.681788 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.681708    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c033dd-33b5-4002-8161-d8a9402c7cd4-kube-api-access-576nm" pod "45c033dd-33b5-4002-8161-d8a9402c7cd4" (UID: "45c033dd-33b5-4002-8161-d8a9402c7cd4"). InnerVolumeSpecName "kube-api-access-576nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:12.681788 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.681752    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-token" pod "45c033dd-33b5-4002-8161-d8a9402c7cd4" (UID: "45c033dd-33b5-4002-8161-d8a9402c7cd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:12.681861 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.681826    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-envoy" pod "45c033dd-33b5-4002-8161-d8a9402c7cd4" (UID: "45c033dd-33b5-4002-8161-d8a9402c7cd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:12.781839 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.781807    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:12.781839 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.781840    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-576nm\" (UniqueName: \"kubernetes.io/projected/45c033dd-33b5-4002-8161-d8a9402c7cd4-kube-api-access-576nm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:12.782055 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.781855    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:12.782055 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:12.781867    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45c033dd-33b5-4002-8161-d8a9402c7cd4-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:13.314715 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:13.314685    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:23:13.375559 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:13.375529    4553 generic.go:356] "Generic (PLEG): container finished" podID="45c033dd-33b5-4002-8161-d8a9402c7cd4" containerID="17dad28906f86142c6d81d85e3b379df72eb47d2288303f28b55b40bd0ef9d92" exitCode=0
Oct 30 22:23:13.375747 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:13.375585    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"
Oct 30 22:23:13.375747 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:13.375616    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2" event={"ID":"45c033dd-33b5-4002-8161-d8a9402c7cd4","Type":"ContainerDied","Data":"17dad28906f86142c6d81d85e3b379df72eb47d2288303f28b55b40bd0ef9d92"}
Oct 30 22:23:13.375747 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:13.375672    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2" event={"ID":"45c033dd-33b5-4002-8161-d8a9402c7cd4","Type":"ContainerDied","Data":"b075f567b01ca614ca885e83630fd478303d5ac39abe46d120be1127a18babfe"}
Oct 30 22:23:13.375747 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:13.375694    4553 scope.go:122] "RemoveContainer" containerID="17dad28906f86142c6d81d85e3b379df72eb47d2288303f28b55b40bd0ef9d92"
Oct 30 22:23:13.386459 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:13.386441    4553 scope.go:122] "RemoveContainer" containerID="011b3d302d36733cec9d1af32f2b3b52cffc145751e8349301f8ee911c544198"
Oct 30 22:23:13.396681 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:13.396662    4553 scope.go:122] "RemoveContainer" containerID="17dad28906f86142c6d81d85e3b379df72eb47d2288303f28b55b40bd0ef9d92"
Oct 30 22:23:13.396940 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:23:13.396923    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17dad28906f86142c6d81d85e3b379df72eb47d2288303f28b55b40bd0ef9d92\": container with ID starting with 17dad28906f86142c6d81d85e3b379df72eb47d2288303f28b55b40bd0ef9d92 not found: ID does not exist" containerID="17dad28906f86142c6d81d85e3b379df72eb47d2288303f28b55b40bd0ef9d92"
Oct 30 22:23:13.397014 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:13.396949    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17dad28906f86142c6d81d85e3b379df72eb47d2288303f28b55b40bd0ef9d92"} err="failed to get container status \"17dad28906f86142c6d81d85e3b379df72eb47d2288303f28b55b40bd0ef9d92\": rpc error: code = NotFound desc = could not find container \"17dad28906f86142c6d81d85e3b379df72eb47d2288303f28b55b40bd0ef9d92\": container with ID starting with 17dad28906f86142c6d81d85e3b379df72eb47d2288303f28b55b40bd0ef9d92 not found: ID does not exist"
Oct 30 22:23:13.397014 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:13.396966    4553 scope.go:122] "RemoveContainer" containerID="011b3d302d36733cec9d1af32f2b3b52cffc145751e8349301f8ee911c544198"
Oct 30 22:23:13.397327 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:23:13.397311    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011b3d302d36733cec9d1af32f2b3b52cffc145751e8349301f8ee911c544198\": container with ID starting with 011b3d302d36733cec9d1af32f2b3b52cffc145751e8349301f8ee911c544198 not found: ID does not exist" containerID="011b3d302d36733cec9d1af32f2b3b52cffc145751e8349301f8ee911c544198"
Oct 30 22:23:13.397372 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:13.397331    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011b3d302d36733cec9d1af32f2b3b52cffc145751e8349301f8ee911c544198"} err="failed to get container status \"011b3d302d36733cec9d1af32f2b3b52cffc145751e8349301f8ee911c544198\": rpc error: code = NotFound desc = could not find container \"011b3d302d36733cec9d1af32f2b3b52cffc145751e8349301f8ee911c544198\": container with ID starting with 011b3d302d36733cec9d1af32f2b3b52cffc145751e8349301f8ee911c544198 not found: ID does not exist"
Oct 30 22:23:13.549350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:13.549308    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"]
Oct 30 22:23:13.552472 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:13.552440    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a28730-openshift-default-5dd7449995-8zvk2"]
Oct 30 22:23:15.395082 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:15.395048    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45c033dd-33b5-4002-8161-d8a9402c7cd4" path="/var/lib/kubelet/pods/45c033dd-33b5-4002-8161-d8a9402c7cd4/volumes"
Oct 30 22:23:16.242549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.242503    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"]
Oct 30 22:23:16.242777 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.242752    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f" podUID="27966b9b-3178-4164-8446-1115ad22399c" containerName="maas-api" containerID="cri-o://f332381de5f06906405868ddb81f81a1be846b6dee74ee058a06492b1f85b915" gracePeriod=30
Oct 30 22:23:16.400842 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.400803    4553 generic.go:356] "Generic (PLEG): container finished" podID="27966b9b-3178-4164-8446-1115ad22399c" containerID="f332381de5f06906405868ddb81f81a1be846b6dee74ee058a06492b1f85b915" exitCode=0
Oct 30 22:23:16.401330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.400879    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f" event={"ID":"27966b9b-3178-4164-8446-1115ad22399c","Type":"ContainerDied","Data":"f332381de5f06906405868ddb81f81a1be846b6dee74ee058a06492b1f85b915"}
Oct 30 22:23:16.559284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.559252    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:23:16.732751 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.732715    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27966b9b-3178-4164-8446-1115ad22399c-kube-api-access-qk9vv\" (UniqueName: \"kubernetes.io/projected/27966b9b-3178-4164-8446-1115ad22399c-kube-api-access-qk9vv\") pod \"27966b9b-3178-4164-8446-1115ad22399c\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") "
Oct 30 22:23:16.732955 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.732860    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27966b9b-3178-4164-8446-1115ad22399c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27966b9b-3178-4164-8446-1115ad22399c-openshift-service-ca\") pod \"27966b9b-3178-4164-8446-1115ad22399c\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") "
Oct 30 22:23:16.733020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.732977    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-maas-api-tls\") pod \"27966b9b-3178-4164-8446-1115ad22399c\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") "
Oct 30 22:23:16.733078 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.733039    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-metrics-certs\") pod \"27966b9b-3178-4164-8446-1115ad22399c\" (UID: \"27966b9b-3178-4164-8446-1115ad22399c\") "
Oct 30 22:23:16.733262 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.733235    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27966b9b-3178-4164-8446-1115ad22399c-openshift-service-ca" pod "27966b9b-3178-4164-8446-1115ad22399c" (UID: "27966b9b-3178-4164-8446-1115ad22399c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:16.733692 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.733671    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27966b9b-3178-4164-8446-1115ad22399c-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:16.735426 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.735397    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27966b9b-3178-4164-8446-1115ad22399c-kube-api-access-qk9vv" pod "27966b9b-3178-4164-8446-1115ad22399c" (UID: "27966b9b-3178-4164-8446-1115ad22399c"). InnerVolumeSpecName "kube-api-access-qk9vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:16.735726 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.735642    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-maas-api-tls" pod "27966b9b-3178-4164-8446-1115ad22399c" (UID: "27966b9b-3178-4164-8446-1115ad22399c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:16.735901 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.735874    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-metrics-certs" pod "27966b9b-3178-4164-8446-1115ad22399c" (UID: "27966b9b-3178-4164-8446-1115ad22399c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:16.797120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.797032    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"]
Oct 30 22:23:16.834792 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.834761    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:16.834792 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.834790    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27966b9b-3178-4164-8446-1115ad22399c-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:16.834792 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.834801    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk9vv\" (UniqueName: \"kubernetes.io/projected/27966b9b-3178-4164-8446-1115ad22399c-kube-api-access-qk9vv\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:16.909936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:16.909899    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"]
Oct 30 22:23:17.419607 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:17.419572    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f" event={"ID":"27966b9b-3178-4164-8446-1115ad22399c","Type":"ContainerDied","Data":"9f85911c216021810999f5e503caf91ac85d92b948fcd15c949a71141d66be11"}
Oct 30 22:23:17.420218 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:17.420198    4553 scope.go:122] "RemoveContainer" containerID="f332381de5f06906405868ddb81f81a1be846b6dee74ee058a06492b1f85b915"
Oct 30 22:23:17.420482 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:17.420466    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"
Oct 30 22:23:17.593844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:17.593800    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"]
Oct 30 22:23:17.598174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:17.598109    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d66ae6-7b94495577-87v8f"]
Oct 30 22:23:19.395511 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:19.395476    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27966b9b-3178-4164-8446-1115ad22399c" path="/var/lib/kubelet/pods/27966b9b-3178-4164-8446-1115ad22399c/volumes"
Oct 30 22:23:21.797480 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:21.797415    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl" podUID="65182276-f498-4e0a-b778-46315447c3dd" containerName="payload-processing" containerID="cri-o://8778fa6dad24532011d777e30a34418ba5c561507d2cf0aeb881a0186aa0e3cb" gracePeriod=25
Oct 30 22:23:21.910974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:21.910915    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2" podUID="c16f4fde-a1b5-41a4-b045-b71cc29d500f" containerName="payload-pre-processing" containerID="cri-o://fc57e93bb3953182bebe9288432e22d68aa4d70e453c5ab4dfacbbe4d56a9b43" gracePeriod=25
Oct 30 22:23:22.073151 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.073117    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"
Oct 30 22:23:22.177532 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.177509    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"
Oct 30 22:23:22.204620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.204591    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65182276-f498-4e0a-b778-46315447c3dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65182276-f498-4e0a-b778-46315447c3dd-plugins-config-volume\") pod \"65182276-f498-4e0a-b778-46315447c3dd\" (UID: \"65182276-f498-4e0a-b778-46315447c3dd\") "
Oct 30 22:23:22.204802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.204640    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65182276-f498-4e0a-b778-46315447c3dd-kube-api-access-vmqwx\" (UniqueName: \"kubernetes.io/projected/65182276-f498-4e0a-b778-46315447c3dd-kube-api-access-vmqwx\") pod \"65182276-f498-4e0a-b778-46315447c3dd\" (UID: \"65182276-f498-4e0a-b778-46315447c3dd\") "
Oct 30 22:23:22.205076 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.205048    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65182276-f498-4e0a-b778-46315447c3dd-plugins-config-volume" pod "65182276-f498-4e0a-b778-46315447c3dd" (UID: "65182276-f498-4e0a-b778-46315447c3dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:22.205433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.205411    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65182276-f498-4e0a-b778-46315447c3dd-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:22.207199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.207169    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65182276-f498-4e0a-b778-46315447c3dd-kube-api-access-vmqwx" pod "65182276-f498-4e0a-b778-46315447c3dd" (UID: "65182276-f498-4e0a-b778-46315447c3dd"). InnerVolumeSpecName "kube-api-access-vmqwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:22.306361 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.306264    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c16f4fde-a1b5-41a4-b045-b71cc29d500f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c16f4fde-a1b5-41a4-b045-b71cc29d500f-plugins-config-volume\") pod \"c16f4fde-a1b5-41a4-b045-b71cc29d500f\" (UID: \"c16f4fde-a1b5-41a4-b045-b71cc29d500f\") "
Oct 30 22:23:22.306361 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.306312    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16f4fde-a1b5-41a4-b045-b71cc29d500f-kube-api-access-hp6wz\" (UniqueName: \"kubernetes.io/projected/c16f4fde-a1b5-41a4-b045-b71cc29d500f-kube-api-access-hp6wz\") pod \"c16f4fde-a1b5-41a4-b045-b71cc29d500f\" (UID: \"c16f4fde-a1b5-41a4-b045-b71cc29d500f\") "
Oct 30 22:23:22.306707 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.306680    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c16f4fde-a1b5-41a4-b045-b71cc29d500f-plugins-config-volume" pod "c16f4fde-a1b5-41a4-b045-b71cc29d500f" (UID: "c16f4fde-a1b5-41a4-b045-b71cc29d500f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:22.306997 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.306977    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmqwx\" (UniqueName: \"kubernetes.io/projected/65182276-f498-4e0a-b778-46315447c3dd-kube-api-access-vmqwx\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:22.307106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.307003    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c16f4fde-a1b5-41a4-b045-b71cc29d500f-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:22.309236 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.309015    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16f4fde-a1b5-41a4-b045-b71cc29d500f-kube-api-access-hp6wz" pod "c16f4fde-a1b5-41a4-b045-b71cc29d500f" (UID: "c16f4fde-a1b5-41a4-b045-b71cc29d500f"). InnerVolumeSpecName "kube-api-access-hp6wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:22.408394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.408358    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp6wz\" (UniqueName: \"kubernetes.io/projected/c16f4fde-a1b5-41a4-b045-b71cc29d500f-kube-api-access-hp6wz\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:22.455310 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.455279    4553 generic.go:356] "Generic (PLEG): container finished" podID="65182276-f498-4e0a-b778-46315447c3dd" containerID="8778fa6dad24532011d777e30a34418ba5c561507d2cf0aeb881a0186aa0e3cb" exitCode=0
Oct 30 22:23:22.455474 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.455333    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"
Oct 30 22:23:22.455474 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.455377    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl" event={"ID":"65182276-f498-4e0a-b778-46315447c3dd","Type":"ContainerDied","Data":"8778fa6dad24532011d777e30a34418ba5c561507d2cf0aeb881a0186aa0e3cb"}
Oct 30 22:23:22.455474 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.455437    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl" event={"ID":"65182276-f498-4e0a-b778-46315447c3dd","Type":"ContainerDied","Data":"0ec0f8d755e2439bf02e428787e23e24e1f5bbf96e4a093e91098c5a6c6ddf60"}
Oct 30 22:23:22.455474 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.455469    4553 scope.go:122] "RemoveContainer" containerID="8778fa6dad24532011d777e30a34418ba5c561507d2cf0aeb881a0186aa0e3cb"
Oct 30 22:23:22.456911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.456888    4553 generic.go:356] "Generic (PLEG): container finished" podID="c16f4fde-a1b5-41a4-b045-b71cc29d500f" containerID="fc57e93bb3953182bebe9288432e22d68aa4d70e453c5ab4dfacbbe4d56a9b43" exitCode=0
Oct 30 22:23:22.457015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.456934    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"
Oct 30 22:23:22.457015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.456981    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2" event={"ID":"c16f4fde-a1b5-41a4-b045-b71cc29d500f","Type":"ContainerDied","Data":"fc57e93bb3953182bebe9288432e22d68aa4d70e453c5ab4dfacbbe4d56a9b43"}
Oct 30 22:23:22.457117 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.457036    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2" event={"ID":"c16f4fde-a1b5-41a4-b045-b71cc29d500f","Type":"ContainerDied","Data":"363d9a30615dd0d8f4357f0f57e331255db8bd09dd6ad66a6f46416f9df7dad5"}
Oct 30 22:23:22.467440 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.467419    4553 scope.go:122] "RemoveContainer" containerID="8778fa6dad24532011d777e30a34418ba5c561507d2cf0aeb881a0186aa0e3cb"
Oct 30 22:23:22.467772 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:23:22.467753    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8778fa6dad24532011d777e30a34418ba5c561507d2cf0aeb881a0186aa0e3cb\": container with ID starting with 8778fa6dad24532011d777e30a34418ba5c561507d2cf0aeb881a0186aa0e3cb not found: ID does not exist" containerID="8778fa6dad24532011d777e30a34418ba5c561507d2cf0aeb881a0186aa0e3cb"
Oct 30 22:23:22.467832 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.467779    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8778fa6dad24532011d777e30a34418ba5c561507d2cf0aeb881a0186aa0e3cb"} err="failed to get container status \"8778fa6dad24532011d777e30a34418ba5c561507d2cf0aeb881a0186aa0e3cb\": rpc error: code = NotFound desc = could not find container \"8778fa6dad24532011d777e30a34418ba5c561507d2cf0aeb881a0186aa0e3cb\": container with ID starting with 8778fa6dad24532011d777e30a34418ba5c561507d2cf0aeb881a0186aa0e3cb not found: ID does not exist"
Oct 30 22:23:22.467832 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.467796    4553 scope.go:122] "RemoveContainer" containerID="fc57e93bb3953182bebe9288432e22d68aa4d70e453c5ab4dfacbbe4d56a9b43"
Oct 30 22:23:22.477713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.477693    4553 scope.go:122] "RemoveContainer" containerID="fc57e93bb3953182bebe9288432e22d68aa4d70e453c5ab4dfacbbe4d56a9b43"
Oct 30 22:23:22.478005 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:23:22.477986    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc57e93bb3953182bebe9288432e22d68aa4d70e453c5ab4dfacbbe4d56a9b43\": container with ID starting with fc57e93bb3953182bebe9288432e22d68aa4d70e453c5ab4dfacbbe4d56a9b43 not found: ID does not exist" containerID="fc57e93bb3953182bebe9288432e22d68aa4d70e453c5ab4dfacbbe4d56a9b43"
Oct 30 22:23:22.478052 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.478014    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc57e93bb3953182bebe9288432e22d68aa4d70e453c5ab4dfacbbe4d56a9b43"} err="failed to get container status \"fc57e93bb3953182bebe9288432e22d68aa4d70e453c5ab4dfacbbe4d56a9b43\": rpc error: code = NotFound desc = could not find container \"fc57e93bb3953182bebe9288432e22d68aa4d70e453c5ab4dfacbbe4d56a9b43\": container with ID starting with fc57e93bb3953182bebe9288432e22d68aa4d70e453c5ab4dfacbbe4d56a9b43 not found: ID does not exist"
Oct 30 22:23:22.628251 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.628222    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"]
Oct 30 22:23:22.631434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.631412    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d66ae6-6c9cbfb979-ltrg2"]
Oct 30 22:23:22.741424 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.741392    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"]
Oct 30 22:23:22.747284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:22.747246    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d66ae6-754d795479-px7fl"]
Oct 30 22:23:23.394149 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:23.394105    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65182276-f498-4e0a-b778-46315447c3dd" path="/var/lib/kubelet/pods/65182276-f498-4e0a-b778-46315447c3dd/volumes"
Oct 30 22:23:23.394630 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:23.394613    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c16f4fde-a1b5-41a4-b045-b71cc29d500f" path="/var/lib/kubelet/pods/c16f4fde-a1b5-41a4-b045-b71cc29d500f/volumes"
Oct 30 22:23:27.377106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377065    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"]
Oct 30 22:23:27.377689 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377674    4553 cpu_manager.go:414] "containerMap: removing container" podUID="45c033dd-33b5-4002-8161-d8a9402c7cd4" containerName="istio-proxy"
Oct 30 22:23:27.377743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377690    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c033dd-33b5-4002-8161-d8a9402c7cd4" containerName="istio-proxy"
Oct 30 22:23:27.377743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377708    4553 cpu_manager.go:414] "containerMap: removing container" podUID="c16f4fde-a1b5-41a4-b045-b71cc29d500f" containerName="payload-pre-processing"
Oct 30 22:23:27.377743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377713    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16f4fde-a1b5-41a4-b045-b71cc29d500f" containerName="payload-pre-processing"
Oct 30 22:23:27.377743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377722    4553 cpu_manager.go:414] "containerMap: removing container" podUID="27966b9b-3178-4164-8446-1115ad22399c" containerName="maas-api"
Oct 30 22:23:27.377743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377727    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27966b9b-3178-4164-8446-1115ad22399c" containerName="maas-api"
Oct 30 22:23:27.377743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377737    4553 cpu_manager.go:414] "containerMap: removing container" podUID="65182276-f498-4e0a-b778-46315447c3dd" containerName="payload-processing"
Oct 30 22:23:27.377743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377741    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65182276-f498-4e0a-b778-46315447c3dd" containerName="payload-processing"
Oct 30 22:23:27.377977 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377747    4553 cpu_manager.go:414] "containerMap: removing container" podUID="45c033dd-33b5-4002-8161-d8a9402c7cd4" containerName="istio-proxy"
Oct 30 22:23:27.377977 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377751    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c033dd-33b5-4002-8161-d8a9402c7cd4" containerName="istio-proxy"
Oct 30 22:23:27.377977 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377833    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16f4fde-a1b5-41a4-b045-b71cc29d500f" containerName="payload-pre-processing"
Oct 30 22:23:27.377977 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377839    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="27966b9b-3178-4164-8446-1115ad22399c" containerName="maas-api"
Oct 30 22:23:27.377977 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377848    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="65182276-f498-4e0a-b778-46315447c3dd" containerName="payload-processing"
Oct 30 22:23:27.377977 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377854    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c033dd-33b5-4002-8161-d8a9402c7cd4" containerName="istio-proxy"
Oct 30 22:23:27.377977 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.377865    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c033dd-33b5-4002-8161-d8a9402c7cd4" containerName="istio-proxy"
Oct 30 22:23:27.381363 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.381345    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"
Oct 30 22:23:27.438622 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.438590    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"]
Oct 30 22:23:27.521937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.521896    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-cce50a\""
Oct 30 22:23:27.522110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.522067    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-cce50a-dockercfg-2st55\""
Oct 30 22:23:27.528062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.522760    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f440cc-1942-4ccd-92d6-936b1c59ac30-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk\" (UID: \"48f440cc-1942-4ccd-92d6-936b1c59ac30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"
Oct 30 22:23:27.528062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.523005    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w726\" (UniqueName: \"kubernetes.io/projected/48f440cc-1942-4ccd-92d6-936b1c59ac30-kube-api-access-7w726\") pod \"payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk\" (UID: \"48f440cc-1942-4ccd-92d6-936b1c59ac30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"
Oct 30 22:23:27.623850 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.623818    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f440cc-1942-4ccd-92d6-936b1c59ac30-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk\" (UID: \"48f440cc-1942-4ccd-92d6-936b1c59ac30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"
Oct 30 22:23:27.624020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.623982    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w726\" (UniqueName: \"kubernetes.io/projected/48f440cc-1942-4ccd-92d6-936b1c59ac30-kube-api-access-7w726\") pod \"payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk\" (UID: \"48f440cc-1942-4ccd-92d6-936b1c59ac30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"
Oct 30 22:23:27.624809 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.624784    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f440cc-1942-4ccd-92d6-936b1c59ac30-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk\" (UID: \"48f440cc-1942-4ccd-92d6-936b1c59ac30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"
Oct 30 22:23:27.683264 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.683175    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w726\" (UniqueName: \"kubernetes.io/projected/48f440cc-1942-4ccd-92d6-936b1c59ac30-kube-api-access-7w726\") pod \"payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk\" (UID: \"48f440cc-1942-4ccd-92d6-936b1c59ac30\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"
Oct 30 22:23:27.696037 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:27.696005    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"
Oct 30 22:23:28.075704 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:28.075676    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"]
Oct 30 22:23:28.077336 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:23:28.077310    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f440cc_1942_4ccd_92d6_936b1c59ac30.slice/crio-2c70e9daadfb5321f7f86f61ecdc4483bbada8a22248fca643e8bc8b767e25de WatchSource:0}: Error finding container 2c70e9daadfb5321f7f86f61ecdc4483bbada8a22248fca643e8bc8b767e25de: Status 404 returned error can't find the container with id 2c70e9daadfb5321f7f86f61ecdc4483bbada8a22248fca643e8bc8b767e25de
Oct 30 22:23:28.202012 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:23:28.201977    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:23:28.541376 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:28.541342    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk" event={"ID":"48f440cc-1942-4ccd-92d6-936b1c59ac30","Type":"ContainerStarted","Data":"d1796f69cdaff3bfc4a33e2e154994fa5c65128a2015de02b32584c1fe1911d7"}
Oct 30 22:23:28.541806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:28.541405    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk" event={"ID":"48f440cc-1942-4ccd-92d6-936b1c59ac30","Type":"ContainerStarted","Data":"2c70e9daadfb5321f7f86f61ecdc4483bbada8a22248fca643e8bc8b767e25de"}
Oct 30 22:23:28.541806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:28.541435    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"
Oct 30 22:23:28.668202 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:28.668154    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk" podStartSLOduration=1.668123603 podStartE2EDuration="1.668123603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:28.666432098 +0000 UTC m=+2459.400795130" watchObservedRunningTime="2026-10-30 22:23:28.668123603 +0000 UTC m=+2459.402486635"
Oct 30 22:23:39.566499 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:39.566465    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"
Oct 30 22:23:46.551059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:46.551022    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"]
Oct 30 22:23:46.661943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:46.661909    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"]
Oct 30 22:23:51.551842 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:51.551783    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs" podUID="a6108731-ab6a-416d-909b-662dba26de39" containerName="payload-processing" containerID="cri-o://f843d5f6b54043574c533ce72d5ee7c1b1d4bf83f21b9d9d50acd20d218aa328" gracePeriod=25
Oct 30 22:23:51.662706 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:51.662668    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24" podUID="3212edbe-040d-489b-8e26-7371e06cbacd" containerName="payload-pre-processing" containerID="cri-o://6e168c04f425124cf21f10a9bd3327d043110b155258a15c00e8b23f5af0d36d" gracePeriod=25
Oct 30 22:23:51.695877 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:51.695845    4553 generic.go:356] "Generic (PLEG): container finished" podID="a6108731-ab6a-416d-909b-662dba26de39" containerID="f843d5f6b54043574c533ce72d5ee7c1b1d4bf83f21b9d9d50acd20d218aa328" exitCode=0
Oct 30 22:23:51.696061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:51.695918    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs" event={"ID":"a6108731-ab6a-416d-909b-662dba26de39","Type":"ContainerDied","Data":"f843d5f6b54043574c533ce72d5ee7c1b1d4bf83f21b9d9d50acd20d218aa328"}
Oct 30 22:23:51.856832 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:51.856805    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"
Oct 30 22:23:51.978529 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:51.978505    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"
Oct 30 22:23:52.049468 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.049428    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6108731-ab6a-416d-909b-662dba26de39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6108731-ab6a-416d-909b-662dba26de39-plugins-config-volume\") pod \"a6108731-ab6a-416d-909b-662dba26de39\" (UID: \"a6108731-ab6a-416d-909b-662dba26de39\") "
Oct 30 22:23:52.049676 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.049489    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6108731-ab6a-416d-909b-662dba26de39-kube-api-access-rmlcf\" (UniqueName: \"kubernetes.io/projected/a6108731-ab6a-416d-909b-662dba26de39-kube-api-access-rmlcf\") pod \"a6108731-ab6a-416d-909b-662dba26de39\" (UID: \"a6108731-ab6a-416d-909b-662dba26de39\") "
Oct 30 22:23:52.049866 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.049843    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6108731-ab6a-416d-909b-662dba26de39-plugins-config-volume" pod "a6108731-ab6a-416d-909b-662dba26de39" (UID: "a6108731-ab6a-416d-909b-662dba26de39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:52.050013 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.049988    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6108731-ab6a-416d-909b-662dba26de39-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:52.051825 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.051803    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6108731-ab6a-416d-909b-662dba26de39-kube-api-access-rmlcf" pod "a6108731-ab6a-416d-909b-662dba26de39" (UID: "a6108731-ab6a-416d-909b-662dba26de39"). InnerVolumeSpecName "kube-api-access-rmlcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:52.150430 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.150400    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3212edbe-040d-489b-8e26-7371e06cbacd-kube-api-access-mx6gc\" (UniqueName: \"kubernetes.io/projected/3212edbe-040d-489b-8e26-7371e06cbacd-kube-api-access-mx6gc\") pod \"3212edbe-040d-489b-8e26-7371e06cbacd\" (UID: \"3212edbe-040d-489b-8e26-7371e06cbacd\") "
Oct 30 22:23:52.150645 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.150441    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3212edbe-040d-489b-8e26-7371e06cbacd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3212edbe-040d-489b-8e26-7371e06cbacd-plugins-config-volume\") pod \"3212edbe-040d-489b-8e26-7371e06cbacd\" (UID: \"3212edbe-040d-489b-8e26-7371e06cbacd\") "
Oct 30 22:23:52.150950 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.150921    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3212edbe-040d-489b-8e26-7371e06cbacd-plugins-config-volume" pod "3212edbe-040d-489b-8e26-7371e06cbacd" (UID: "3212edbe-040d-489b-8e26-7371e06cbacd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:52.151110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.151090    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3212edbe-040d-489b-8e26-7371e06cbacd-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:52.151183 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.151118    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmlcf\" (UniqueName: \"kubernetes.io/projected/a6108731-ab6a-416d-909b-662dba26de39-kube-api-access-rmlcf\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:52.153192 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.153154    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3212edbe-040d-489b-8e26-7371e06cbacd-kube-api-access-mx6gc" pod "3212edbe-040d-489b-8e26-7371e06cbacd" (UID: "3212edbe-040d-489b-8e26-7371e06cbacd"). InnerVolumeSpecName "kube-api-access-mx6gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:52.252802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.252762    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx6gc\" (UniqueName: \"kubernetes.io/projected/3212edbe-040d-489b-8e26-7371e06cbacd-kube-api-access-mx6gc\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:23:52.703970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.703942    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"
Oct 30 22:23:52.703970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.703961    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs" event={"ID":"a6108731-ab6a-416d-909b-662dba26de39","Type":"ContainerDied","Data":"4e8dca81ab3e915d3212dacbb83cea8ee36ef057423d79b95e1491cd683e54ec"}
Oct 30 22:23:52.704551 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.704025    4553 scope.go:122] "RemoveContainer" containerID="f843d5f6b54043574c533ce72d5ee7c1b1d4bf83f21b9d9d50acd20d218aa328"
Oct 30 22:23:52.705623 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.705600    4553 generic.go:356] "Generic (PLEG): container finished" podID="3212edbe-040d-489b-8e26-7371e06cbacd" containerID="6e168c04f425124cf21f10a9bd3327d043110b155258a15c00e8b23f5af0d36d" exitCode=0
Oct 30 22:23:52.705709 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.705652    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"
Oct 30 22:23:52.705709 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.705689    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24" event={"ID":"3212edbe-040d-489b-8e26-7371e06cbacd","Type":"ContainerDied","Data":"6e168c04f425124cf21f10a9bd3327d043110b155258a15c00e8b23f5af0d36d"}
Oct 30 22:23:52.705834 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.705749    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24" event={"ID":"3212edbe-040d-489b-8e26-7371e06cbacd","Type":"ContainerDied","Data":"490eb588cfd97d0c6845e57047764f8880deb7b0b3bb586ded2dc19f1ff34c9a"}
Oct 30 22:23:52.714855 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.714839    4553 scope.go:122] "RemoveContainer" containerID="6e168c04f425124cf21f10a9bd3327d043110b155258a15c00e8b23f5af0d36d"
Oct 30 22:23:52.724496 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.724472    4553 scope.go:122] "RemoveContainer" containerID="6e168c04f425124cf21f10a9bd3327d043110b155258a15c00e8b23f5af0d36d"
Oct 30 22:23:52.724824 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:23:52.724803    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e168c04f425124cf21f10a9bd3327d043110b155258a15c00e8b23f5af0d36d\": container with ID starting with 6e168c04f425124cf21f10a9bd3327d043110b155258a15c00e8b23f5af0d36d not found: ID does not exist" containerID="6e168c04f425124cf21f10a9bd3327d043110b155258a15c00e8b23f5af0d36d"
Oct 30 22:23:52.724884 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.724834    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e168c04f425124cf21f10a9bd3327d043110b155258a15c00e8b23f5af0d36d"} err="failed to get container status \"6e168c04f425124cf21f10a9bd3327d043110b155258a15c00e8b23f5af0d36d\": rpc error: code = NotFound desc = could not find container \"6e168c04f425124cf21f10a9bd3327d043110b155258a15c00e8b23f5af0d36d\": container with ID starting with 6e168c04f425124cf21f10a9bd3327d043110b155258a15c00e8b23f5af0d36d not found: ID does not exist"
Oct 30 22:23:52.871985 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.871938    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"]
Oct 30 22:23:52.880256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.880073    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fe2565-6f586c5cdc-wlf24"]
Oct 30 22:23:52.992010 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.991976    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"]
Oct 30 22:23:52.994526 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:52.994501    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fe2565-76d8bdbb7-bhkvs"]
Oct 30 22:23:53.397711 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:53.397680    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3212edbe-040d-489b-8e26-7371e06cbacd" path="/var/lib/kubelet/pods/3212edbe-040d-489b-8e26-7371e06cbacd/volumes"
Oct 30 22:23:53.398117 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:23:53.398095    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6108731-ab6a-416d-909b-662dba26de39" path="/var/lib/kubelet/pods/a6108731-ab6a-416d-909b-662dba26de39/volumes"
Oct 30 22:23:58.201914 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:23:58.201875    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:24:00.837362 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:00.837328    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"]
Oct 30 22:24:00.837935 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:00.837587    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj" podUID="94e79eee-aaaf-49d0-b354-748bdfbe636e" containerName="payload-processing" containerID="cri-o://0ff162e07497aa213aa412f3ef811ea529323b8ff4eb850ef4f83512d9ecba8e" gracePeriod=30
Oct 30 22:24:00.947168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:00.946786    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"]
Oct 30 22:24:00.947168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:00.947046    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs" podUID="0e8c351f-703e-45db-a083-15d13b3399e4" containerName="payload-pre-processing" containerID="cri-o://8d3cb77cf7c5b86d4a21da0757b84593b474c53d5aec5af608ed2567e1b2b373" gracePeriod=30
Oct 30 22:24:01.123337 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.123309    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"
Oct 30 22:24:01.167809 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.167770    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e79eee-aaaf-49d0-b354-748bdfbe636e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e79eee-aaaf-49d0-b354-748bdfbe636e-plugins-config-volume\") pod \"94e79eee-aaaf-49d0-b354-748bdfbe636e\" (UID: \"94e79eee-aaaf-49d0-b354-748bdfbe636e\") "
Oct 30 22:24:01.168003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.167882    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e79eee-aaaf-49d0-b354-748bdfbe636e-kube-api-access-hqvj8\" (UniqueName: \"kubernetes.io/projected/94e79eee-aaaf-49d0-b354-748bdfbe636e-kube-api-access-hqvj8\") pod \"94e79eee-aaaf-49d0-b354-748bdfbe636e\" (UID: \"94e79eee-aaaf-49d0-b354-748bdfbe636e\") "
Oct 30 22:24:01.168239 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.168212    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e79eee-aaaf-49d0-b354-748bdfbe636e-plugins-config-volume" pod "94e79eee-aaaf-49d0-b354-748bdfbe636e" (UID: "94e79eee-aaaf-49d0-b354-748bdfbe636e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:01.170194 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.170167    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e79eee-aaaf-49d0-b354-748bdfbe636e-kube-api-access-hqvj8" pod "94e79eee-aaaf-49d0-b354-748bdfbe636e" (UID: "94e79eee-aaaf-49d0-b354-748bdfbe636e"). InnerVolumeSpecName "kube-api-access-hqvj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:01.220926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.220898    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"
Oct 30 22:24:01.269793 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.269739    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8c351f-703e-45db-a083-15d13b3399e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8c351f-703e-45db-a083-15d13b3399e4-plugins-config-volume\") pod \"0e8c351f-703e-45db-a083-15d13b3399e4\" (UID: \"0e8c351f-703e-45db-a083-15d13b3399e4\") "
Oct 30 22:24:01.269986 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.269885    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8c351f-703e-45db-a083-15d13b3399e4-kube-api-access-frwqf\" (UniqueName: \"kubernetes.io/projected/0e8c351f-703e-45db-a083-15d13b3399e4-kube-api-access-frwqf\") pod \"0e8c351f-703e-45db-a083-15d13b3399e4\" (UID: \"0e8c351f-703e-45db-a083-15d13b3399e4\") "
Oct 30 22:24:01.270068 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.269994    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8c351f-703e-45db-a083-15d13b3399e4-plugins-config-volume" pod "0e8c351f-703e-45db-a083-15d13b3399e4" (UID: "0e8c351f-703e-45db-a083-15d13b3399e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:01.270459 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.270438    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94e79eee-aaaf-49d0-b354-748bdfbe636e-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:01.270542 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.270464    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqvj8\" (UniqueName: \"kubernetes.io/projected/94e79eee-aaaf-49d0-b354-748bdfbe636e-kube-api-access-hqvj8\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:01.270542 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.270481    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8c351f-703e-45db-a083-15d13b3399e4-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:01.272280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.272256    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8c351f-703e-45db-a083-15d13b3399e4-kube-api-access-frwqf" pod "0e8c351f-703e-45db-a083-15d13b3399e4" (UID: "0e8c351f-703e-45db-a083-15d13b3399e4"). InnerVolumeSpecName "kube-api-access-frwqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:01.384324 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.378072    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frwqf\" (UniqueName: \"kubernetes.io/projected/0e8c351f-703e-45db-a083-15d13b3399e4-kube-api-access-frwqf\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:01.798012 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.797915    4553 generic.go:356] "Generic (PLEG): container finished" podID="94e79eee-aaaf-49d0-b354-748bdfbe636e" containerID="0ff162e07497aa213aa412f3ef811ea529323b8ff4eb850ef4f83512d9ecba8e" exitCode=0
Oct 30 22:24:01.798012 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.797966    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"
Oct 30 22:24:01.798328 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.798005    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj" event={"ID":"94e79eee-aaaf-49d0-b354-748bdfbe636e","Type":"ContainerDied","Data":"0ff162e07497aa213aa412f3ef811ea529323b8ff4eb850ef4f83512d9ecba8e"}
Oct 30 22:24:01.798328 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.798065    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj" event={"ID":"94e79eee-aaaf-49d0-b354-748bdfbe636e","Type":"ContainerDied","Data":"05951a4115c0e046482048e0fdb8d8c4c1833ae108d3e5fa367f3c0c5bbfa874"}
Oct 30 22:24:01.798328 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.798098    4553 scope.go:122] "RemoveContainer" containerID="0ff162e07497aa213aa412f3ef811ea529323b8ff4eb850ef4f83512d9ecba8e"
Oct 30 22:24:01.799880 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.799853    4553 generic.go:356] "Generic (PLEG): container finished" podID="0e8c351f-703e-45db-a083-15d13b3399e4" containerID="8d3cb77cf7c5b86d4a21da0757b84593b474c53d5aec5af608ed2567e1b2b373" exitCode=0
Oct 30 22:24:01.800025 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.799913    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"
Oct 30 22:24:01.800025 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.800009    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs" event={"ID":"0e8c351f-703e-45db-a083-15d13b3399e4","Type":"ContainerDied","Data":"8d3cb77cf7c5b86d4a21da0757b84593b474c53d5aec5af608ed2567e1b2b373"}
Oct 30 22:24:01.800164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.800048    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs" event={"ID":"0e8c351f-703e-45db-a083-15d13b3399e4","Type":"ContainerDied","Data":"1665640df7d4b4f9f55b3004b02959e880b351e9994be1701663d5da29efa8f5"}
Oct 30 22:24:01.811045 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.811025    4553 scope.go:122] "RemoveContainer" containerID="0ff162e07497aa213aa412f3ef811ea529323b8ff4eb850ef4f83512d9ecba8e"
Oct 30 22:24:01.811431 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:24:01.811408    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff162e07497aa213aa412f3ef811ea529323b8ff4eb850ef4f83512d9ecba8e\": container with ID starting with 0ff162e07497aa213aa412f3ef811ea529323b8ff4eb850ef4f83512d9ecba8e not found: ID does not exist" containerID="0ff162e07497aa213aa412f3ef811ea529323b8ff4eb850ef4f83512d9ecba8e"
Oct 30 22:24:01.811519 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.811439    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff162e07497aa213aa412f3ef811ea529323b8ff4eb850ef4f83512d9ecba8e"} err="failed to get container status \"0ff162e07497aa213aa412f3ef811ea529323b8ff4eb850ef4f83512d9ecba8e\": rpc error: code = NotFound desc = could not find container \"0ff162e07497aa213aa412f3ef811ea529323b8ff4eb850ef4f83512d9ecba8e\": container with ID starting with 0ff162e07497aa213aa412f3ef811ea529323b8ff4eb850ef4f83512d9ecba8e not found: ID does not exist"
Oct 30 22:24:01.811519 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.811458    4553 scope.go:122] "RemoveContainer" containerID="8d3cb77cf7c5b86d4a21da0757b84593b474c53d5aec5af608ed2567e1b2b373"
Oct 30 22:24:01.820917 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.820894    4553 scope.go:122] "RemoveContainer" containerID="8d3cb77cf7c5b86d4a21da0757b84593b474c53d5aec5af608ed2567e1b2b373"
Oct 30 22:24:01.821303 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:24:01.821283    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d3cb77cf7c5b86d4a21da0757b84593b474c53d5aec5af608ed2567e1b2b373\": container with ID starting with 8d3cb77cf7c5b86d4a21da0757b84593b474c53d5aec5af608ed2567e1b2b373 not found: ID does not exist" containerID="8d3cb77cf7c5b86d4a21da0757b84593b474c53d5aec5af608ed2567e1b2b373"
Oct 30 22:24:01.821370 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.821312    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d3cb77cf7c5b86d4a21da0757b84593b474c53d5aec5af608ed2567e1b2b373"} err="failed to get container status \"8d3cb77cf7c5b86d4a21da0757b84593b474c53d5aec5af608ed2567e1b2b373\": rpc error: code = NotFound desc = could not find container \"8d3cb77cf7c5b86d4a21da0757b84593b474c53d5aec5af608ed2567e1b2b373\": container with ID starting with 8d3cb77cf7c5b86d4a21da0757b84593b474c53d5aec5af608ed2567e1b2b373 not found: ID does not exist"
Oct 30 22:24:01.966853 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.966819    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"]
Oct 30 22:24:01.971113 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:01.971085    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2edb49-78d986d578-bxdkj"]
Oct 30 22:24:02.080703 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:02.080616    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"]
Oct 30 22:24:02.084512 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:02.084483    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2edb49-65d8969d87-knsxs"]
Oct 30 22:24:02.672883 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:02.672853    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"]
Oct 30 22:24:02.673089 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:02.673028    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68" podUID="1b459720-eccc-40bb-ad95-3032719cc96c" containerName="discovery" containerID="cri-o://f9eacde1e10aece24887a50a4cbe5000ea0850af6d91cc7ae17a01b53b62dad0" gracePeriod=30
Oct 30 22:24:02.809471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:02.809441    4553 generic.go:356] "Generic (PLEG): container finished" podID="1b459720-eccc-40bb-ad95-3032719cc96c" containerID="f9eacde1e10aece24887a50a4cbe5000ea0850af6d91cc7ae17a01b53b62dad0" exitCode=0
Oct 30 22:24:02.809668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:02.809512    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68" event={"ID":"1b459720-eccc-40bb-ad95-3032719cc96c","Type":"ContainerDied","Data":"f9eacde1e10aece24887a50a4cbe5000ea0850af6d91cc7ae17a01b53b62dad0"}
Oct 30 22:24:02.956752 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:02.956717    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 22:24:03.101880 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.101842    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b459720-eccc-40bb-ad95-3032719cc96c-istio-token\" (UniqueName: \"kubernetes.io/projected/1b459720-eccc-40bb-ad95-3032719cc96c-istio-token\") pod \"1b459720-eccc-40bb-ad95-3032719cc96c\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") "
Oct 30 22:24:03.102416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.101968    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-istio-kubeconfig\") pod \"1b459720-eccc-40bb-ad95-3032719cc96c\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") "
Oct 30 22:24:03.102416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.102010    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-istio-csr-dns-cert\") pod \"1b459720-eccc-40bb-ad95-3032719cc96c\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") "
Oct 30 22:24:03.102416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.102110    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b459720-eccc-40bb-ad95-3032719cc96c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b459720-eccc-40bb-ad95-3032719cc96c-local-certs\") pod \"1b459720-eccc-40bb-ad95-3032719cc96c\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") "
Oct 30 22:24:03.102416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.102159    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b459720-eccc-40bb-ad95-3032719cc96c-kube-api-access-844xm\" (UniqueName: \"kubernetes.io/projected/1b459720-eccc-40bb-ad95-3032719cc96c-kube-api-access-844xm\") pod \"1b459720-eccc-40bb-ad95-3032719cc96c\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") "
Oct 30 22:24:03.102416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.102222    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b459720-eccc-40bb-ad95-3032719cc96c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b459720-eccc-40bb-ad95-3032719cc96c-istio-csr-ca-configmap\") pod \"1b459720-eccc-40bb-ad95-3032719cc96c\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") "
Oct 30 22:24:03.102416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.102254    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-cacerts\" (UniqueName: \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-cacerts\") pod \"1b459720-eccc-40bb-ad95-3032719cc96c\" (UID: \"1b459720-eccc-40bb-ad95-3032719cc96c\") "
Oct 30 22:24:03.103236 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.103191    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b459720-eccc-40bb-ad95-3032719cc96c-istio-csr-ca-configmap" pod "1b459720-eccc-40bb-ad95-3032719cc96c" (UID: "1b459720-eccc-40bb-ad95-3032719cc96c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:03.105479 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.105426    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-istio-kubeconfig" pod "1b459720-eccc-40bb-ad95-3032719cc96c" (UID: "1b459720-eccc-40bb-ad95-3032719cc96c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:03.105605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.105535    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b459720-eccc-40bb-ad95-3032719cc96c-kube-api-access-844xm" pod "1b459720-eccc-40bb-ad95-3032719cc96c" (UID: "1b459720-eccc-40bb-ad95-3032719cc96c"). InnerVolumeSpecName "kube-api-access-844xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:03.105823 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.105759    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-istio-csr-dns-cert" pod "1b459720-eccc-40bb-ad95-3032719cc96c" (UID: "1b459720-eccc-40bb-ad95-3032719cc96c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:03.106028 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.106000    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-cacerts" pod "1b459720-eccc-40bb-ad95-3032719cc96c" (UID: "1b459720-eccc-40bb-ad95-3032719cc96c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:03.107454 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.107426    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b459720-eccc-40bb-ad95-3032719cc96c-local-certs" pod "1b459720-eccc-40bb-ad95-3032719cc96c" (UID: "1b459720-eccc-40bb-ad95-3032719cc96c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:03.107571 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.107431    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b459720-eccc-40bb-ad95-3032719cc96c-istio-token" pod "1b459720-eccc-40bb-ad95-3032719cc96c" (UID: "1b459720-eccc-40bb-ad95-3032719cc96c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:03.204641 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.204557    4553 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b459720-eccc-40bb-ad95-3032719cc96c-local-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:03.204641 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.204586    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-844xm\" (UniqueName: \"kubernetes.io/projected/1b459720-eccc-40bb-ad95-3032719cc96c-kube-api-access-844xm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:03.204641 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.204601    4553 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1b459720-eccc-40bb-ad95-3032719cc96c-istio-csr-ca-configmap\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:03.204641 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.204612    4553 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-cacerts\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:03.204641 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.204626    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b459720-eccc-40bb-ad95-3032719cc96c-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:03.204641 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.204637    4553 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-istio-kubeconfig\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:03.204641 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.204647    4553 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1b459720-eccc-40bb-ad95-3032719cc96c-istio-csr-dns-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:03.392984 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.392949    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8c351f-703e-45db-a083-15d13b3399e4" path="/var/lib/kubelet/pods/0e8c351f-703e-45db-a083-15d13b3399e4/volumes"
Oct 30 22:24:03.393412 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.393398    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e79eee-aaaf-49d0-b354-748bdfbe636e" path="/var/lib/kubelet/pods/94e79eee-aaaf-49d0-b354-748bdfbe636e/volumes"
Oct 30 22:24:03.817593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.817561    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68" event={"ID":"1b459720-eccc-40bb-ad95-3032719cc96c","Type":"ContainerDied","Data":"78211fbc6b61ec4fa851918335a5e6c5f9b193d855a85905bddda7d3236a8587"}
Oct 30 22:24:03.817593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.817597    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"
Oct 30 22:24:03.817850 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:03.817618    4553 scope.go:122] "RemoveContainer" containerID="f9eacde1e10aece24887a50a4cbe5000ea0850af6d91cc7ae17a01b53b62dad0"
Oct 30 22:24:04.036175 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:04.036127    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"]
Oct 30 22:24:04.040663 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:04.040633    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q5v68"]
Oct 30 22:24:05.393631 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:05.393597    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b459720-eccc-40bb-ad95-3032719cc96c" path="/var/lib/kubelet/pods/1b459720-eccc-40bb-ad95-3032719cc96c/volumes"
Oct 30 22:24:10.610937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:10.610900    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"]
Oct 30 22:24:10.611406 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:10.611117    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk" podUID="8e3d7161-a39d-4501-a8e5-6f547c91237b" containerName="istio-proxy" containerID="cri-o://d6460913d5290a32170d6c7cf8466f146a1b08b4c282020d7d15d6b496dc4bbb" gracePeriod=30
Oct 30 22:24:15.885922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.885855    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 22:24:15.895713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.895685    4553 generic.go:356] "Generic (PLEG): container finished" podID="8e3d7161-a39d-4501-a8e5-6f547c91237b" containerID="d6460913d5290a32170d6c7cf8466f146a1b08b4c282020d7d15d6b496dc4bbb" exitCode=0
Oct 30 22:24:15.895858 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.895732    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"
Oct 30 22:24:15.895858 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.895767    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk" event={"ID":"8e3d7161-a39d-4501-a8e5-6f547c91237b","Type":"ContainerDied","Data":"d6460913d5290a32170d6c7cf8466f146a1b08b4c282020d7d15d6b496dc4bbb"}
Oct 30 22:24:15.895858 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.895829    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk" event={"ID":"8e3d7161-a39d-4501-a8e5-6f547c91237b","Type":"ContainerDied","Data":"062607f81e27b9d2238ad31ceeeb97c0ac19e0a162677542858e258fde77da18"}
Oct 30 22:24:15.895967 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.895857    4553 scope.go:122] "RemoveContainer" containerID="d6460913d5290a32170d6c7cf8466f146a1b08b4c282020d7d15d6b496dc4bbb"
Oct 30 22:24:15.905317 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.905290    4553 scope.go:122] "RemoveContainer" containerID="d6460913d5290a32170d6c7cf8466f146a1b08b4c282020d7d15d6b496dc4bbb"
Oct 30 22:24:15.905599 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:24:15.905571    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6460913d5290a32170d6c7cf8466f146a1b08b4c282020d7d15d6b496dc4bbb\": container with ID starting with d6460913d5290a32170d6c7cf8466f146a1b08b4c282020d7d15d6b496dc4bbb not found: ID does not exist" containerID="d6460913d5290a32170d6c7cf8466f146a1b08b4c282020d7d15d6b496dc4bbb"
Oct 30 22:24:15.905693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.905610    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6460913d5290a32170d6c7cf8466f146a1b08b4c282020d7d15d6b496dc4bbb"} err="failed to get container status \"d6460913d5290a32170d6c7cf8466f146a1b08b4c282020d7d15d6b496dc4bbb\": rpc error: code = NotFound desc = could not find container \"d6460913d5290a32170d6c7cf8466f146a1b08b4c282020d7d15d6b496dc4bbb\": container with ID starting with d6460913d5290a32170d6c7cf8466f146a1b08b4c282020d7d15d6b496dc4bbb not found: ID does not exist"
Oct 30 22:24:15.974682 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.974636    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-data\") pod \"8e3d7161-a39d-4501-a8e5-6f547c91237b\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") "
Oct 30 22:24:15.974682 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.974686    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3d7161-a39d-4501-a8e5-6f547c91237b-kube-api-access-sbqmh\" (UniqueName: \"kubernetes.io/projected/8e3d7161-a39d-4501-a8e5-6f547c91237b-kube-api-access-sbqmh\") pod \"8e3d7161-a39d-4501-a8e5-6f547c91237b\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") "
Oct 30 22:24:15.974936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.974762    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-credential-socket\") pod \"8e3d7161-a39d-4501-a8e5-6f547c91237b\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") "
Oct 30 22:24:15.974936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.974779    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-envoy\") pod \"8e3d7161-a39d-4501-a8e5-6f547c91237b\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") "
Oct 30 22:24:15.974936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.974860    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-token\" (UniqueName: \"kubernetes.io/projected/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-token\") pod \"8e3d7161-a39d-4501-a8e5-6f547c91237b\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") "
Oct 30 22:24:15.974936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.974913    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3d7161-a39d-4501-a8e5-6f547c91237b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3d7161-a39d-4501-a8e5-6f547c91237b-istiod-ca-cert\") pod \"8e3d7161-a39d-4501-a8e5-6f547c91237b\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") "
Oct 30 22:24:15.975213 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.974940    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-podinfo\") pod \"8e3d7161-a39d-4501-a8e5-6f547c91237b\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") "
Oct 30 22:24:15.975213 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.974973    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-workload-certs\") pod \"8e3d7161-a39d-4501-a8e5-6f547c91237b\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") "
Oct 30 22:24:15.975213 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.974986    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-data" pod "8e3d7161-a39d-4501-a8e5-6f547c91237b" (UID: "8e3d7161-a39d-4501-a8e5-6f547c91237b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:15.975213 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.975071    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-workload-socket\") pod \"8e3d7161-a39d-4501-a8e5-6f547c91237b\" (UID: \"8e3d7161-a39d-4501-a8e5-6f547c91237b\") "
Oct 30 22:24:15.975213 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.975123    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-credential-socket" pod "8e3d7161-a39d-4501-a8e5-6f547c91237b" (UID: "8e3d7161-a39d-4501-a8e5-6f547c91237b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:15.975524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.975319    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-workload-certs" pod "8e3d7161-a39d-4501-a8e5-6f547c91237b" (UID: "8e3d7161-a39d-4501-a8e5-6f547c91237b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:15.975524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.975406    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3d7161-a39d-4501-a8e5-6f547c91237b-istiod-ca-cert" pod "8e3d7161-a39d-4501-a8e5-6f547c91237b" (UID: "8e3d7161-a39d-4501-a8e5-6f547c91237b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:15.975524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.975429    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-workload-socket" pod "8e3d7161-a39d-4501-a8e5-6f547c91237b" (UID: "8e3d7161-a39d-4501-a8e5-6f547c91237b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:15.975861 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.975837    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3d7161-a39d-4501-a8e5-6f547c91237b-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:15.975934 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.975866    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:15.975934 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.975882    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:15.975934 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.975896    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:15.975934 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.975910    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:15.977257 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.977225    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3d7161-a39d-4501-a8e5-6f547c91237b-kube-api-access-sbqmh" pod "8e3d7161-a39d-4501-a8e5-6f547c91237b" (UID: "8e3d7161-a39d-4501-a8e5-6f547c91237b"). InnerVolumeSpecName "kube-api-access-sbqmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:15.977553 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.977538    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-envoy" pod "8e3d7161-a39d-4501-a8e5-6f547c91237b" (UID: "8e3d7161-a39d-4501-a8e5-6f547c91237b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:15.977833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.977801    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-podinfo" pod "8e3d7161-a39d-4501-a8e5-6f547c91237b" (UID: "8e3d7161-a39d-4501-a8e5-6f547c91237b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:24:15.977833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:15.977813    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-token" pod "8e3d7161-a39d-4501-a8e5-6f547c91237b" (UID: "8e3d7161-a39d-4501-a8e5-6f547c91237b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:16.076664 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:16.076624    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbqmh\" (UniqueName: \"kubernetes.io/projected/8e3d7161-a39d-4501-a8e5-6f547c91237b-kube-api-access-sbqmh\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:16.076664 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:16.076657    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:16.076664 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:16.076670    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:16.076994 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:16.076683    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3d7161-a39d-4501-a8e5-6f547c91237b-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:16.322088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:16.322050    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"]
Oct 30 22:24:16.328845 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:16.328810    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-x9kqk"]
Oct 30 22:24:16.762054 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:16.762015    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"]
Oct 30 22:24:16.762265 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:16.762242    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy" containerID="cri-o://09adb8417253ca932ad9f4ff3d73181299ed2519c9df3acd9a3f38393ed11493" gracePeriod=30
Oct 30 22:24:17.396352 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:17.394563    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e3d7161-a39d-4501-a8e5-6f547c91237b" path="/var/lib/kubelet/pods/8e3d7161-a39d-4501-a8e5-6f547c91237b/volumes"
Oct 30 22:24:18.291602 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.291568    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"]
Oct 30 22:24:18.292116 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292103    4553 cpu_manager.go:414] "containerMap: removing container" podUID="8e3d7161-a39d-4501-a8e5-6f547c91237b" containerName="istio-proxy"
Oct 30 22:24:18.292203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292118    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3d7161-a39d-4501-a8e5-6f547c91237b" containerName="istio-proxy"
Oct 30 22:24:18.292203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292162    4553 cpu_manager.go:414] "containerMap: removing container" podUID="0e8c351f-703e-45db-a083-15d13b3399e4" containerName="payload-pre-processing"
Oct 30 22:24:18.292203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292168    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8c351f-703e-45db-a083-15d13b3399e4" containerName="payload-pre-processing"
Oct 30 22:24:18.292203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292178    4553 cpu_manager.go:414] "containerMap: removing container" podUID="94e79eee-aaaf-49d0-b354-748bdfbe636e" containerName="payload-processing"
Oct 30 22:24:18.292203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292183    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e79eee-aaaf-49d0-b354-748bdfbe636e" containerName="payload-processing"
Oct 30 22:24:18.292203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292193    4553 cpu_manager.go:414] "containerMap: removing container" podUID="1b459720-eccc-40bb-ad95-3032719cc96c" containerName="discovery"
Oct 30 22:24:18.292203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292197    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b459720-eccc-40bb-ad95-3032719cc96c" containerName="discovery"
Oct 30 22:24:18.292423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292207    4553 cpu_manager.go:414] "containerMap: removing container" podUID="3212edbe-040d-489b-8e26-7371e06cbacd" containerName="payload-pre-processing"
Oct 30 22:24:18.292423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292212    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3212edbe-040d-489b-8e26-7371e06cbacd" containerName="payload-pre-processing"
Oct 30 22:24:18.292423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292230    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a6108731-ab6a-416d-909b-662dba26de39" containerName="payload-processing"
Oct 30 22:24:18.292423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292234    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6108731-ab6a-416d-909b-662dba26de39" containerName="payload-processing"
Oct 30 22:24:18.292423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292310    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8c351f-703e-45db-a083-15d13b3399e4" containerName="payload-pre-processing"
Oct 30 22:24:18.292423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292319    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e79eee-aaaf-49d0-b354-748bdfbe636e" containerName="payload-processing"
Oct 30 22:24:18.292423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292325    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b459720-eccc-40bb-ad95-3032719cc96c" containerName="discovery"
Oct 30 22:24:18.292423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292330    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="3212edbe-040d-489b-8e26-7371e06cbacd" containerName="payload-pre-processing"
Oct 30 22:24:18.292423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292337    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6108731-ab6a-416d-909b-662dba26de39" containerName="payload-processing"
Oct 30 22:24:18.292423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.292344    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3d7161-a39d-4501-a8e5-6f547c91237b" containerName="istio-proxy"
Oct 30 22:24:18.297111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.297089    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.351288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.351254    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-37a390-openshift-default-dockercfg-z76cz\""
Oct 30 22:24:18.405192 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.405151    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"]
Oct 30 22:24:18.405776 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.405750    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-credential-socket\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.405886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.405801    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2af516-6dad-461a-9674-8951b4231d49-istio-podinfo\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.405950 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.405900    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2af516-6dad-461a-9674-8951b4231d49-istiod-ca-cert\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.406003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.405951    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-istio-data\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.406094 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.406080    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-istio-envoy\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.406204 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.406147    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2af516-6dad-461a-9674-8951b4231d49-istio-token\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.406285 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.406215    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-workload-socket\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.406342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.406292    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xthz\" (UniqueName: \"kubernetes.io/projected/7a2af516-6dad-461a-9674-8951b4231d49-kube-api-access-5xthz\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.406342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.406325    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-workload-certs\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.507218 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.507182    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-credential-socket\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.507417 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.507251    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2af516-6dad-461a-9674-8951b4231d49-istio-podinfo\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.507417 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.507287    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2af516-6dad-461a-9674-8951b4231d49-istiod-ca-cert\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.507417 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.507334    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-istio-data\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.507580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.507436    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-istio-envoy\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.507580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.507502    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2af516-6dad-461a-9674-8951b4231d49-istio-token\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.507580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.507528    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-workload-socket\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.507580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.507566    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xthz\" (UniqueName: \"kubernetes.io/projected/7a2af516-6dad-461a-9674-8951b4231d49-kube-api-access-5xthz\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.507785 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.507595    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-workload-certs\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.507785 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.507669    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-credential-socket\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.507964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.507935    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-istio-data\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.508117 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.508030    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-workload-certs\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.508239 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.508196    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-workload-socket\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.508508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.508487    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2af516-6dad-461a-9674-8951b4231d49-istiod-ca-cert\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.509811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.509791    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-istio-envoy\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.510041 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.510024    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2af516-6dad-461a-9674-8951b4231d49-istio-podinfo\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.567824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.567748    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2af516-6dad-461a-9674-8951b4231d49-istio-token\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.567824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.567791    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xthz\" (UniqueName: \"kubernetes.io/projected/7a2af516-6dad-461a-9674-8951b4231d49-kube-api-access-5xthz\") pod \"e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") " pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:18.611062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:18.611008    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:19.048856 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:19.048827    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"]
Oct 30 22:24:19.049806 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:24:19.049784    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2af516_6dad_461a_9674_8951b4231d49.slice/crio-f99757da4d90836061e3163c2c5a6cf5c22d2f867272d6b71589ab3fd867bcdf WatchSource:0}: Error finding container f99757da4d90836061e3163c2c5a6cf5c22d2f867272d6b71589ab3fd867bcdf: Status 404 returned error can't find the container with id f99757da4d90836061e3163c2c5a6cf5c22d2f867272d6b71589ab3fd867bcdf
Oct 30 22:24:19.055682 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:19.055642    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:24:19.055765 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:19.055731    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:24:19.055800 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:19.055774    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:24:19.926390 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:19.926346    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l" event={"ID":"7a2af516-6dad-461a-9674-8951b4231d49","Type":"ContainerStarted","Data":"1293b95ef6bfaf6824859b78f419990577047219114765459a00efb46c2d61f7"}
Oct 30 22:24:19.926813 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:19.926417    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l" event={"ID":"7a2af516-6dad-461a-9674-8951b4231d49","Type":"ContainerStarted","Data":"f99757da4d90836061e3163c2c5a6cf5c22d2f867272d6b71589ab3fd867bcdf"}
Oct 30 22:24:20.151338 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:20.151282    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l" podStartSLOduration=2.151261634 podStartE2EDuration="2.151261634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:20.099031111 +0000 UTC m=+2510.833394156" watchObservedRunningTime="2026-10-30 22:24:20.151261634 +0000 UTC m=+2510.885624664"
Oct 30 22:24:20.611752 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:20.611722    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:20.617929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:20.617901    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:20.933258 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:20.933167    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:20.934127 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:20.934109    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:24:21.955246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:21.955207    4553 generic.go:356] "Generic (PLEG): container finished" podID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerID="09adb8417253ca932ad9f4ff3d73181299ed2519c9df3acd9a3f38393ed11493" exitCode=0
Oct 30 22:24:21.955630 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:21.955284    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" event={"ID":"c2f1880e-2463-4df9-9c57-1d0dabe76c4b","Type":"ContainerDied","Data":"09adb8417253ca932ad9f4ff3d73181299ed2519c9df3acd9a3f38393ed11493"}
Oct 30 22:24:22.036476 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.036401    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:24:22.152663 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.152625    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-credential-socket\") pod \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") "
Oct 30 22:24:22.152663 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.152667    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istiod-ca-cert\") pod \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") "
Oct 30 22:24:22.152904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.152735    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-envoy\") pod \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") "
Oct 30 22:24:22.152904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.152871    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-workload-certs\") pod \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") "
Oct 30 22:24:22.152984 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.152928    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-credential-socket" pod "c2f1880e-2463-4df9-9c57-1d0dabe76c4b" (UID: "c2f1880e-2463-4df9-9c57-1d0dabe76c4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:22.152984 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.152967    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-data\") pod \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") "
Oct 30 22:24:22.153081 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.153063    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-podinfo\") pod \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") "
Oct 30 22:24:22.153166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.153109    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-token\" (UniqueName: \"kubernetes.io/projected/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-token\") pod \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") "
Oct 30 22:24:22.153238 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.153196    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-kube-api-access-pgk5x\" (UniqueName: \"kubernetes.io/projected/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-kube-api-access-pgk5x\") pod \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") "
Oct 30 22:24:22.153238 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.153198    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-workload-certs" pod "c2f1880e-2463-4df9-9c57-1d0dabe76c4b" (UID: "c2f1880e-2463-4df9-9c57-1d0dabe76c4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:22.153238 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.153209    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-data" pod "c2f1880e-2463-4df9-9c57-1d0dabe76c4b" (UID: "c2f1880e-2463-4df9-9c57-1d0dabe76c4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:22.153399 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.153240    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istiod-ca-cert" pod "c2f1880e-2463-4df9-9c57-1d0dabe76c4b" (UID: "c2f1880e-2463-4df9-9c57-1d0dabe76c4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:22.153399 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.153246    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-workload-socket\") pod \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\" (UID: \"c2f1880e-2463-4df9-9c57-1d0dabe76c4b\") "
Oct 30 22:24:22.153582 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.153558    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-workload-socket" pod "c2f1880e-2463-4df9-9c57-1d0dabe76c4b" (UID: "c2f1880e-2463-4df9-9c57-1d0dabe76c4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:22.153857 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.153770    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.153857 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.153793    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.153857 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.153806    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.153857 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.153819    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.153857 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.153831    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.155384 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.155364    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-envoy" pod "c2f1880e-2463-4df9-9c57-1d0dabe76c4b" (UID: "c2f1880e-2463-4df9-9c57-1d0dabe76c4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:22.155491 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.155470    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-podinfo" pod "c2f1880e-2463-4df9-9c57-1d0dabe76c4b" (UID: "c2f1880e-2463-4df9-9c57-1d0dabe76c4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:24:22.155936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.155918    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-token" pod "c2f1880e-2463-4df9-9c57-1d0dabe76c4b" (UID: "c2f1880e-2463-4df9-9c57-1d0dabe76c4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:22.155999 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.155944    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-kube-api-access-pgk5x" pod "c2f1880e-2463-4df9-9c57-1d0dabe76c4b" (UID: "c2f1880e-2463-4df9-9c57-1d0dabe76c4b"). InnerVolumeSpecName "kube-api-access-pgk5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:22.255354 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.255325    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.255354 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.255350    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.255354 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.255359    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgk5x\" (UniqueName: \"kubernetes.io/projected/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-kube-api-access-pgk5x\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.255600 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.255369    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2f1880e-2463-4df9-9c57-1d0dabe76c4b-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.963658 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.963629    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"
Oct 30 22:24:22.964094 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.963664    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2" event={"ID":"c2f1880e-2463-4df9-9c57-1d0dabe76c4b","Type":"ContainerDied","Data":"73309d2b766f33805e67a480eb7693d22c73dbe251e9ab550f5e4743c69abcbd"}
Oct 30 22:24:22.964094 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:22.963734    4553 scope.go:122] "RemoveContainer" containerID="09adb8417253ca932ad9f4ff3d73181299ed2519c9df3acd9a3f38393ed11493"
Oct 30 22:24:23.138760 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:23.138721    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"]
Oct 30 22:24:23.189087 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:23.189053    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1c3eef-openshift-default-7c9cd69cb9-76vf2"]
Oct 30 22:24:23.392943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:23.392904    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" path="/var/lib/kubelet/pods/c2f1880e-2463-4df9-9c57-1d0dabe76c4b/volumes"
Oct 30 22:24:24.546040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.546005    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"]
Oct 30 22:24:24.546619 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.546604    4553 cpu_manager.go:414] "containerMap: removing container" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy"
Oct 30 22:24:24.546667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.546620    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy"
Oct 30 22:24:24.546744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.546735    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2f1880e-2463-4df9-9c57-1d0dabe76c4b" containerName="istio-proxy"
Oct 30 22:24:24.550182 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.550158    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:24.608371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.608292    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-cce50a-serving-cert\""
Oct 30 22:24:24.661751 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.661716    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"]
Oct 30 22:24:24.683752 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.683707    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-metrics-certs\") pod \"maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:24.683925 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.683759    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hcgk\" (UniqueName: \"kubernetes.io/projected/6bbfe405-12a5-4eae-851a-8b2fc99947a0-kube-api-access-6hcgk\") pod \"maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:24.684007 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.683958    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:24.684007 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.683989    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bbfe405-12a5-4eae-851a-8b2fc99947a0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:24.785837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.785800    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-metrics-certs\") pod \"maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:24.785837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.785840    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcgk\" (UniqueName: \"kubernetes.io/projected/6bbfe405-12a5-4eae-851a-8b2fc99947a0-kube-api-access-6hcgk\") pod \"maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:24.786030 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.785951    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:24.786030 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.785980    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bbfe405-12a5-4eae-851a-8b2fc99947a0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:24.786158 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:24:24.786087    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-serving-cert: secret "maas-api-e2e-praxis-a-cce50a-serving-cert" not found
Oct 30 22:24:24.786207 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:24:24.786166    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-maas-api-tls podName:6bbfe405-12a5-4eae-851a-8b2fc99947a0 nodeName:}" failed. No retries permitted until 2026-10-30 22:24:25.286150249 +0000 UTC m=+2516.020513257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-maas-api-tls") pod "maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7" (UID: "6bbfe405-12a5-4eae-851a-8b2fc99947a0") : secret "maas-api-e2e-praxis-a-cce50a-serving-cert" not found
Oct 30 22:24:24.786682 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.786664    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bbfe405-12a5-4eae-851a-8b2fc99947a0-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:24.788537 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.788519    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-metrics-certs\") pod \"maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:24.849421 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:24.849384    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hcgk\" (UniqueName: \"kubernetes.io/projected/6bbfe405-12a5-4eae-851a-8b2fc99947a0-kube-api-access-6hcgk\") pod \"maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:25.292473 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:25.292379    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:25.295059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:25.295031    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-maas-api-tls\") pod \"maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:25.467933 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:25.467893    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:25.851163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:25.851112    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"]
Oct 30 22:24:25.853536 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:24:25.853507    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbfe405_12a5_4eae_851a_8b2fc99947a0.slice/crio-c25a87ded4327a5aafede1352277116922be589b6c44951b7a9d0fcf84ea3262 WatchSource:0}: Error finding container c25a87ded4327a5aafede1352277116922be589b6c44951b7a9d0fcf84ea3262: Status 404 returned error can't find the container with id c25a87ded4327a5aafede1352277116922be589b6c44951b7a9d0fcf84ea3262
Oct 30 22:24:25.986102 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:25.986064    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7" event={"ID":"6bbfe405-12a5-4eae-851a-8b2fc99947a0","Type":"ContainerStarted","Data":"c25a87ded4327a5aafede1352277116922be589b6c44951b7a9d0fcf84ea3262"}
Oct 30 22:24:26.994701 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:26.994667    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7" event={"ID":"6bbfe405-12a5-4eae-851a-8b2fc99947a0","Type":"ContainerStarted","Data":"42298b409eee7424f1c4fc451e5b701a97efdbad7d6c33069d707d2900d04f80"}
Oct 30 22:24:26.995119 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:26.994741    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:27.112582 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:27.112531    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7" podStartSLOduration=2.154129141 podStartE2EDuration="3.112514562s" totalImagesPullingTime="958.385421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:27.111670438 +0000 UTC m=+2517.846033469" watchObservedRunningTime="2026-10-30 22:24:27.112514562 +0000 UTC m=+2517.846877593"
Oct 30 22:24:28.202533 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:24:28.202496    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:24:28.446709 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.446669    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"]
Oct 30 22:24:28.450889 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.450869    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"
Oct 30 22:24:28.508040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.508011    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"]
Oct 30 22:24:28.531873 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.531832    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab37b2d1-4c30-4d6f-971f-96195f6a4d27-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8\" (UID: \"ab37b2d1-4c30-4d6f-971f-96195f6a4d27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"
Oct 30 22:24:28.532346 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.532320    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkc5k\" (UniqueName: \"kubernetes.io/projected/ab37b2d1-4c30-4d6f-971f-96195f6a4d27-kube-api-access-qkc5k\") pod \"payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8\" (UID: \"ab37b2d1-4c30-4d6f-971f-96195f6a4d27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"
Oct 30 22:24:28.555799 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.555669    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-37a390-dockercfg-zf7nl\""
Oct 30 22:24:28.555799 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.555765    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-37a390\""
Oct 30 22:24:28.609502 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.609465    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"]
Oct 30 22:24:28.613604 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.613584    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"
Oct 30 22:24:28.633725 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.633696    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab37b2d1-4c30-4d6f-971f-96195f6a4d27-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8\" (UID: \"ab37b2d1-4c30-4d6f-971f-96195f6a4d27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"
Oct 30 22:24:28.633858 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.633785    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkc5k\" (UniqueName: \"kubernetes.io/projected/ab37b2d1-4c30-4d6f-971f-96195f6a4d27-kube-api-access-qkc5k\") pod \"payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8\" (UID: \"ab37b2d1-4c30-4d6f-971f-96195f6a4d27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"
Oct 30 22:24:28.634452 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.634434    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab37b2d1-4c30-4d6f-971f-96195f6a4d27-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8\" (UID: \"ab37b2d1-4c30-4d6f-971f-96195f6a4d27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"
Oct 30 22:24:28.662684 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.662647    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"]
Oct 30 22:24:28.693029 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.692998    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkc5k\" (UniqueName: \"kubernetes.io/projected/ab37b2d1-4c30-4d6f-971f-96195f6a4d27-kube-api-access-qkc5k\") pod \"payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8\" (UID: \"ab37b2d1-4c30-4d6f-971f-96195f6a4d27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"
Oct 30 22:24:28.734838 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.734747    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8x9b\" (UniqueName: \"kubernetes.io/projected/b1433d5b-9380-414f-a649-82fb38e66302-kube-api-access-s8x9b\") pod \"payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5\" (UID: \"b1433d5b-9380-414f-a649-82fb38e66302\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"
Oct 30 22:24:28.734978 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.734860    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1433d5b-9380-414f-a649-82fb38e66302-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5\" (UID: \"b1433d5b-9380-414f-a649-82fb38e66302\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"
Oct 30 22:24:28.765417 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.765381    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"
Oct 30 22:24:28.838748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.836920    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8x9b\" (UniqueName: \"kubernetes.io/projected/b1433d5b-9380-414f-a649-82fb38e66302-kube-api-access-s8x9b\") pod \"payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5\" (UID: \"b1433d5b-9380-414f-a649-82fb38e66302\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"
Oct 30 22:24:28.838748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.837036    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1433d5b-9380-414f-a649-82fb38e66302-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5\" (UID: \"b1433d5b-9380-414f-a649-82fb38e66302\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"
Oct 30 22:24:28.839603 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.839165    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1433d5b-9380-414f-a649-82fb38e66302-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5\" (UID: \"b1433d5b-9380-414f-a649-82fb38e66302\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"
Oct 30 22:24:28.896854 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.896816    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8x9b\" (UniqueName: \"kubernetes.io/projected/b1433d5b-9380-414f-a649-82fb38e66302-kube-api-access-s8x9b\") pod \"payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5\" (UID: \"b1433d5b-9380-414f-a649-82fb38e66302\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"
Oct 30 22:24:28.927622 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:28.927589    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"
Oct 30 22:24:29.151998 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:29.151950    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"]
Oct 30 22:24:29.159166 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:24:29.159107    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab37b2d1_4c30_4d6f_971f_96195f6a4d27.slice/crio-934a2c2848d5f2a0b7c453ddb800de46c5c505782589153d3f68456ce6209fd3 WatchSource:0}: Error finding container 934a2c2848d5f2a0b7c453ddb800de46c5c505782589153d3f68456ce6209fd3: Status 404 returned error can't find the container with id 934a2c2848d5f2a0b7c453ddb800de46c5c505782589153d3f68456ce6209fd3
Oct 30 22:24:29.304764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:29.304736    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"]
Oct 30 22:24:29.305630 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:24:29.305604    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1433d5b_9380_414f_a649_82fb38e66302.slice/crio-b218a1c6e2fae3953b1eb74d476aedd3d7eaed42a51ad03ec1d399e6125991b3 WatchSource:0}: Error finding container b218a1c6e2fae3953b1eb74d476aedd3d7eaed42a51ad03ec1d399e6125991b3: Status 404 returned error can't find the container with id b218a1c6e2fae3953b1eb74d476aedd3d7eaed42a51ad03ec1d399e6125991b3
Oct 30 22:24:30.018349 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:30.018312    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5" event={"ID":"b1433d5b-9380-414f-a649-82fb38e66302","Type":"ContainerStarted","Data":"a137b2f302f979ab62787e9ca9e267198b24f15e1694c8f1cb20e6f175bb5e1d"}
Oct 30 22:24:30.018555 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:30.018376    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5" event={"ID":"b1433d5b-9380-414f-a649-82fb38e66302","Type":"ContainerStarted","Data":"b218a1c6e2fae3953b1eb74d476aedd3d7eaed42a51ad03ec1d399e6125991b3"}
Oct 30 22:24:30.018555 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:30.018405    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"
Oct 30 22:24:30.020022 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:30.019998    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8" event={"ID":"ab37b2d1-4c30-4d6f-971f-96195f6a4d27","Type":"ContainerStarted","Data":"eb6689edba14badde0285be3c457005673e667c9dc24bf28c41adce1936671fb"}
Oct 30 22:24:30.020149 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:30.020043    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8" event={"ID":"ab37b2d1-4c30-4d6f-971f-96195f6a4d27","Type":"ContainerStarted","Data":"934a2c2848d5f2a0b7c453ddb800de46c5c505782589153d3f68456ce6209fd3"}
Oct 30 22:24:30.020149 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:30.020072    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"
Oct 30 22:24:30.137525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:30.137468    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5" podStartSLOduration=2.137453797 podStartE2EDuration="2.137453797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:30.136309486 +0000 UTC m=+2520.870672520" watchObservedRunningTime="2026-10-30 22:24:30.137453797 +0000 UTC m=+2520.871816828"
Oct 30 22:24:33.008545 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:33.008513    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:24:33.131219 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:33.131164    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8" podStartSLOduration=5.131149458 podStartE2EDuration="5.131149458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:30.253961693 +0000 UTC m=+2520.988324725" watchObservedRunningTime="2026-10-30 22:24:33.131149458 +0000 UTC m=+2523.865512483"
Oct 30 22:24:41.029411 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:41.029383    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"
Oct 30 22:24:41.029877 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:41.029447    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"
Oct 30 22:24:53.289389 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:53.289355    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"]
Oct 30 22:24:53.295353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:53.295325    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"
Oct 30 22:24:53.350840 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:53.350810    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"]
Oct 30 22:24:53.397505 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:53.397472    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-955887\""
Oct 30 22:24:53.397734 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:53.397525    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-955887-dockercfg-nkflt\""
Oct 30 22:24:53.462983 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:53.462951    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkvzz\" (UniqueName: \"kubernetes.io/projected/256e7ef9-77ab-424a-910f-af05d51522a3-kube-api-access-zkvzz\") pod \"payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf\" (UID: \"256e7ef9-77ab-424a-910f-af05d51522a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"
Oct 30 22:24:53.463180 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:53.463012    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256e7ef9-77ab-424a-910f-af05d51522a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf\" (UID: \"256e7ef9-77ab-424a-910f-af05d51522a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"
Oct 30 22:24:53.564168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:53.564032    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkvzz\" (UniqueName: \"kubernetes.io/projected/256e7ef9-77ab-424a-910f-af05d51522a3-kube-api-access-zkvzz\") pod \"payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf\" (UID: \"256e7ef9-77ab-424a-910f-af05d51522a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"
Oct 30 22:24:53.564168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:53.564104    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256e7ef9-77ab-424a-910f-af05d51522a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf\" (UID: \"256e7ef9-77ab-424a-910f-af05d51522a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"
Oct 30 22:24:53.564936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:53.564915    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256e7ef9-77ab-424a-910f-af05d51522a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf\" (UID: \"256e7ef9-77ab-424a-910f-af05d51522a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"
Oct 30 22:24:53.622085 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:53.622050    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkvzz\" (UniqueName: \"kubernetes.io/projected/256e7ef9-77ab-424a-910f-af05d51522a3-kube-api-access-zkvzz\") pod \"payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf\" (UID: \"256e7ef9-77ab-424a-910f-af05d51522a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"
Oct 30 22:24:53.908352 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:53.908318    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"
Oct 30 22:24:54.083812 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:54.083773    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"]
Oct 30 22:24:54.088997 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:54.088974    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"
Oct 30 22:24:54.171937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:54.171856    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"]
Oct 30 22:24:54.272437 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:54.272401    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcc1f853-d639-4df7-b61f-06232e9a6c42-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l\" (UID: \"dcc1f853-d639-4df7-b61f-06232e9a6c42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"
Oct 30 22:24:54.272660 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:54.272463    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfnb4\" (UniqueName: \"kubernetes.io/projected/dcc1f853-d639-4df7-b61f-06232e9a6c42-kube-api-access-rfnb4\") pod \"payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l\" (UID: \"dcc1f853-d639-4df7-b61f-06232e9a6c42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"
Oct 30 22:24:54.285146 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:54.285098    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"]
Oct 30 22:24:54.286765 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:24:54.286737    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256e7ef9_77ab_424a_910f_af05d51522a3.slice/crio-b96a766d856eb1c94213f3146f96a03b9cf874cf6541a05f4dc8e14ab1d1bcb6 WatchSource:0}: Error finding container b96a766d856eb1c94213f3146f96a03b9cf874cf6541a05f4dc8e14ab1d1bcb6: Status 404 returned error can't find the container with id b96a766d856eb1c94213f3146f96a03b9cf874cf6541a05f4dc8e14ab1d1bcb6
Oct 30 22:24:54.373465 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:54.373440    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcc1f853-d639-4df7-b61f-06232e9a6c42-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l\" (UID: \"dcc1f853-d639-4df7-b61f-06232e9a6c42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"
Oct 30 22:24:54.373825 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:54.373503    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfnb4\" (UniqueName: \"kubernetes.io/projected/dcc1f853-d639-4df7-b61f-06232e9a6c42-kube-api-access-rfnb4\") pod \"payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l\" (UID: \"dcc1f853-d639-4df7-b61f-06232e9a6c42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"
Oct 30 22:24:54.374147 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:54.374111    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcc1f853-d639-4df7-b61f-06232e9a6c42-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l\" (UID: \"dcc1f853-d639-4df7-b61f-06232e9a6c42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"
Oct 30 22:24:54.434198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:54.434095    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfnb4\" (UniqueName: \"kubernetes.io/projected/dcc1f853-d639-4df7-b61f-06232e9a6c42-kube-api-access-rfnb4\") pod \"payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l\" (UID: \"dcc1f853-d639-4df7-b61f-06232e9a6c42\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"
Oct 30 22:24:54.703404 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:54.703321    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"
Oct 30 22:24:55.080824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:55.080796    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"]
Oct 30 22:24:55.082327 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:24:55.082296    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc1f853_d639_4df7_b61f_06232e9a6c42.slice/crio-e3b289dd79f58c85eb0f91b4f686c085260a2a6bdbcf23c636b37238ccda6c66 WatchSource:0}: Error finding container e3b289dd79f58c85eb0f91b4f686c085260a2a6bdbcf23c636b37238ccda6c66: Status 404 returned error can't find the container with id e3b289dd79f58c85eb0f91b4f686c085260a2a6bdbcf23c636b37238ccda6c66
Oct 30 22:24:55.177872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:55.177841    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf" event={"ID":"256e7ef9-77ab-424a-910f-af05d51522a3","Type":"ContainerStarted","Data":"39fc54eef9e01df2ee34e360a2d3aeed880b1a41b094e499b5ee8d02d97cd352"}
Oct 30 22:24:55.178067 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:55.177897    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf" event={"ID":"256e7ef9-77ab-424a-910f-af05d51522a3","Type":"ContainerStarted","Data":"b96a766d856eb1c94213f3146f96a03b9cf874cf6541a05f4dc8e14ab1d1bcb6"}
Oct 30 22:24:55.178067 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:55.178033    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"
Oct 30 22:24:55.179744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:55.179718    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l" event={"ID":"dcc1f853-d639-4df7-b61f-06232e9a6c42","Type":"ContainerStarted","Data":"19bdf48fed17919188e12baa4ea6fffad2ccba05591ffb9158bbbbd45de67457"}
Oct 30 22:24:55.179861 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:55.179764    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l" event={"ID":"dcc1f853-d639-4df7-b61f-06232e9a6c42","Type":"ContainerStarted","Data":"e3b289dd79f58c85eb0f91b4f686c085260a2a6bdbcf23c636b37238ccda6c66"}
Oct 30 22:24:55.179861 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:55.179805    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"
Oct 30 22:24:55.295685 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:55.295632    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf" podStartSLOduration=2.295619373 podStartE2EDuration="2.295619373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:55.294382093 +0000 UTC m=+2546.028745123" watchObservedRunningTime="2026-10-30 22:24:55.295619373 +0000 UTC m=+2546.029982404"
Oct 30 22:24:55.420461 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:24:55.420407    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l" podStartSLOduration=1.420391892 podStartE2EDuration="1.420391892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:55.417354479 +0000 UTC m=+2546.151717511" watchObservedRunningTime="2026-10-30 22:24:55.420391892 +0000 UTC m=+2546.154754922"
Oct 30 22:24:58.202800 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:24:58.202769    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:25:06.189884 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:06.189844    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"
Oct 30 22:25:06.190474 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:06.189923    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"
Oct 30 22:25:15.274435 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.274343    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"]
Oct 30 22:25:15.274915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.274614    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh" podUID="a37cba11-3e73-47d7-88c8-133f37788561" containerName="payload-processing" containerID="cri-o://90ee3389baa6a3d8841e2a8390c88c88c1e07b2d515b2912b516dbd271ce7b37" gracePeriod=30
Oct 30 22:25:15.321849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.321811    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"]
Oct 30 22:25:15.322048 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.322025    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g" podUID="ad305e1a-a602-4c0c-9454-f0d783e281fb" containerName="payload-pre-processing" containerID="cri-o://981adbfd97d1c7b1f9ddfb0f60cda5d5971211e67d015157ad3b556c95dfbdc5" gracePeriod=30
Oct 30 22:25:15.603354 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.603320    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"
Oct 30 22:25:15.632678 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.632654    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"
Oct 30 22:25:15.745149 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.745101    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37cba11-3e73-47d7-88c8-133f37788561-kube-api-access-rhvw5\" (UniqueName: \"kubernetes.io/projected/a37cba11-3e73-47d7-88c8-133f37788561-kube-api-access-rhvw5\") pod \"a37cba11-3e73-47d7-88c8-133f37788561\" (UID: \"a37cba11-3e73-47d7-88c8-133f37788561\") "
Oct 30 22:25:15.745347 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.745202    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad305e1a-a602-4c0c-9454-f0d783e281fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad305e1a-a602-4c0c-9454-f0d783e281fb-plugins-config-volume\") pod \"ad305e1a-a602-4c0c-9454-f0d783e281fb\" (UID: \"ad305e1a-a602-4c0c-9454-f0d783e281fb\") "
Oct 30 22:25:15.745347 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.745313    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a37cba11-3e73-47d7-88c8-133f37788561-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37cba11-3e73-47d7-88c8-133f37788561-plugins-config-volume\") pod \"a37cba11-3e73-47d7-88c8-133f37788561\" (UID: \"a37cba11-3e73-47d7-88c8-133f37788561\") "
Oct 30 22:25:15.745483 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.745350    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad305e1a-a602-4c0c-9454-f0d783e281fb-kube-api-access-7kvvp\" (UniqueName: \"kubernetes.io/projected/ad305e1a-a602-4c0c-9454-f0d783e281fb-kube-api-access-7kvvp\") pod \"ad305e1a-a602-4c0c-9454-f0d783e281fb\" (UID: \"ad305e1a-a602-4c0c-9454-f0d783e281fb\") "
Oct 30 22:25:15.745593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.745569    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad305e1a-a602-4c0c-9454-f0d783e281fb-plugins-config-volume" pod "ad305e1a-a602-4c0c-9454-f0d783e281fb" (UID: "ad305e1a-a602-4c0c-9454-f0d783e281fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:15.745833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.745814    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad305e1a-a602-4c0c-9454-f0d783e281fb-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:15.746150 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.746104    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37cba11-3e73-47d7-88c8-133f37788561-plugins-config-volume" pod "a37cba11-3e73-47d7-88c8-133f37788561" (UID: "a37cba11-3e73-47d7-88c8-133f37788561"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:15.747608 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.747585    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37cba11-3e73-47d7-88c8-133f37788561-kube-api-access-rhvw5" pod "a37cba11-3e73-47d7-88c8-133f37788561" (UID: "a37cba11-3e73-47d7-88c8-133f37788561"). InnerVolumeSpecName "kube-api-access-rhvw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:15.747727 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.747706    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad305e1a-a602-4c0c-9454-f0d783e281fb-kube-api-access-7kvvp" pod "ad305e1a-a602-4c0c-9454-f0d783e281fb" (UID: "ad305e1a-a602-4c0c-9454-f0d783e281fb"). InnerVolumeSpecName "kube-api-access-7kvvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:15.847763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.847662    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37cba11-3e73-47d7-88c8-133f37788561-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:15.847763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.847688    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kvvp\" (UniqueName: \"kubernetes.io/projected/ad305e1a-a602-4c0c-9454-f0d783e281fb-kube-api-access-7kvvp\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:15.847763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:15.847698    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhvw5\" (UniqueName: \"kubernetes.io/projected/a37cba11-3e73-47d7-88c8-133f37788561-kube-api-access-rhvw5\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:16.310338 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.310303    4553 generic.go:356] "Generic (PLEG): container finished" podID="a37cba11-3e73-47d7-88c8-133f37788561" containerID="90ee3389baa6a3d8841e2a8390c88c88c1e07b2d515b2912b516dbd271ce7b37" exitCode=0
Oct 30 22:25:16.310841 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.310360    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"
Oct 30 22:25:16.310841 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.310386    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh" event={"ID":"a37cba11-3e73-47d7-88c8-133f37788561","Type":"ContainerDied","Data":"90ee3389baa6a3d8841e2a8390c88c88c1e07b2d515b2912b516dbd271ce7b37"}
Oct 30 22:25:16.310841 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.310442    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh" event={"ID":"a37cba11-3e73-47d7-88c8-133f37788561","Type":"ContainerDied","Data":"8a4b1b20c8ff85212d3499c09027ccae1d45e3cb14a44a4bdf198f87e3a15fec"}
Oct 30 22:25:16.310841 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.310474    4553 scope.go:122] "RemoveContainer" containerID="90ee3389baa6a3d8841e2a8390c88c88c1e07b2d515b2912b516dbd271ce7b37"
Oct 30 22:25:16.312124 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.312098    4553 generic.go:356] "Generic (PLEG): container finished" podID="ad305e1a-a602-4c0c-9454-f0d783e281fb" containerID="981adbfd97d1c7b1f9ddfb0f60cda5d5971211e67d015157ad3b556c95dfbdc5" exitCode=0
Oct 30 22:25:16.312332 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.312178    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g" event={"ID":"ad305e1a-a602-4c0c-9454-f0d783e281fb","Type":"ContainerDied","Data":"981adbfd97d1c7b1f9ddfb0f60cda5d5971211e67d015157ad3b556c95dfbdc5"}
Oct 30 22:25:16.312332 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.312190    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"
Oct 30 22:25:16.312332 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.312224    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g" event={"ID":"ad305e1a-a602-4c0c-9454-f0d783e281fb","Type":"ContainerDied","Data":"cb2767ba87cb908cb5cebc4baa33d5881616047be606c4cacf7d9d6f420c373b"}
Oct 30 22:25:16.322335 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.322317    4553 scope.go:122] "RemoveContainer" containerID="90ee3389baa6a3d8841e2a8390c88c88c1e07b2d515b2912b516dbd271ce7b37"
Oct 30 22:25:16.322643 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:25:16.322622    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ee3389baa6a3d8841e2a8390c88c88c1e07b2d515b2912b516dbd271ce7b37\": container with ID starting with 90ee3389baa6a3d8841e2a8390c88c88c1e07b2d515b2912b516dbd271ce7b37 not found: ID does not exist" containerID="90ee3389baa6a3d8841e2a8390c88c88c1e07b2d515b2912b516dbd271ce7b37"
Oct 30 22:25:16.322740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.322657    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ee3389baa6a3d8841e2a8390c88c88c1e07b2d515b2912b516dbd271ce7b37"} err="failed to get container status \"90ee3389baa6a3d8841e2a8390c88c88c1e07b2d515b2912b516dbd271ce7b37\": rpc error: code = NotFound desc = could not find container \"90ee3389baa6a3d8841e2a8390c88c88c1e07b2d515b2912b516dbd271ce7b37\": container with ID starting with 90ee3389baa6a3d8841e2a8390c88c88c1e07b2d515b2912b516dbd271ce7b37 not found: ID does not exist"
Oct 30 22:25:16.322740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.322685    4553 scope.go:122] "RemoveContainer" containerID="981adbfd97d1c7b1f9ddfb0f60cda5d5971211e67d015157ad3b556c95dfbdc5"
Oct 30 22:25:16.332756 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.332734    4553 scope.go:122] "RemoveContainer" containerID="981adbfd97d1c7b1f9ddfb0f60cda5d5971211e67d015157ad3b556c95dfbdc5"
Oct 30 22:25:16.333035 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:25:16.333013    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981adbfd97d1c7b1f9ddfb0f60cda5d5971211e67d015157ad3b556c95dfbdc5\": container with ID starting with 981adbfd97d1c7b1f9ddfb0f60cda5d5971211e67d015157ad3b556c95dfbdc5 not found: ID does not exist" containerID="981adbfd97d1c7b1f9ddfb0f60cda5d5971211e67d015157ad3b556c95dfbdc5"
Oct 30 22:25:16.333089 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.333045    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981adbfd97d1c7b1f9ddfb0f60cda5d5971211e67d015157ad3b556c95dfbdc5"} err="failed to get container status \"981adbfd97d1c7b1f9ddfb0f60cda5d5971211e67d015157ad3b556c95dfbdc5\": rpc error: code = NotFound desc = could not find container \"981adbfd97d1c7b1f9ddfb0f60cda5d5971211e67d015157ad3b556c95dfbdc5\": container with ID starting with 981adbfd97d1c7b1f9ddfb0f60cda5d5971211e67d015157ad3b556c95dfbdc5 not found: ID does not exist"
Oct 30 22:25:16.477161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.477093    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"]
Oct 30 22:25:16.527741 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.527701    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-96567a-6996946fdc-cwt4g"]
Oct 30 22:25:16.598596 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.598566    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"]
Oct 30 22:25:16.605417 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:16.605390    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-96567a-6f4c8bf5c6-btdxh"]
Oct 30 22:25:17.393313 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:17.393279    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a37cba11-3e73-47d7-88c8-133f37788561" path="/var/lib/kubelet/pods/a37cba11-3e73-47d7-88c8-133f37788561/volumes"
Oct 30 22:25:17.393717 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:17.393682    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad305e1a-a602-4c0c-9454-f0d783e281fb" path="/var/lib/kubelet/pods/ad305e1a-a602-4c0c-9454-f0d783e281fb/volumes"
Oct 30 22:25:21.670299 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.670263    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"]
Oct 30 22:25:21.670895 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.670879    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ad305e1a-a602-4c0c-9454-f0d783e281fb" containerName="payload-pre-processing"
Oct 30 22:25:21.670946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.670897    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad305e1a-a602-4c0c-9454-f0d783e281fb" containerName="payload-pre-processing"
Oct 30 22:25:21.670946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.670914    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a37cba11-3e73-47d7-88c8-133f37788561" containerName="payload-processing"
Oct 30 22:25:21.670946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.670919    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37cba11-3e73-47d7-88c8-133f37788561" containerName="payload-processing"
Oct 30 22:25:21.671042 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.671007    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad305e1a-a602-4c0c-9454-f0d783e281fb" containerName="payload-pre-processing"
Oct 30 22:25:21.671042 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.671020    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37cba11-3e73-47d7-88c8-133f37788561" containerName="payload-processing"
Oct 30 22:25:21.675883 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.675862    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:21.714450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.714409    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-metrics-certs\") pod \"maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:21.714450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.714456    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9060a9ed-94f2-400e-97c4-e3fa723424d1-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:21.714697 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.714577    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-maas-api-tls\") pod \"maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:21.714697 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.714638    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hfg5\" (UniqueName: \"kubernetes.io/projected/9060a9ed-94f2-400e-97c4-e3fa723424d1-kube-api-access-9hfg5\") pod \"maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:21.728862 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.728834    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-37a390-serving-cert\""
Oct 30 22:25:21.782475 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.782445    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"]
Oct 30 22:25:21.815921 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.815887    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-metrics-certs\") pod \"maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:21.816162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.815946    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9060a9ed-94f2-400e-97c4-e3fa723424d1-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:21.816162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.816019    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-maas-api-tls\") pod \"maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:21.816162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.816074    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hfg5\" (UniqueName: \"kubernetes.io/projected/9060a9ed-94f2-400e-97c4-e3fa723424d1-kube-api-access-9hfg5\") pod \"maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:21.816390 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:25:21.816228    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-serving-cert: secret "maas-api-e2e-praxis-b-37a390-serving-cert" not found
Oct 30 22:25:21.816390 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:25:21.816336    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-maas-api-tls podName:9060a9ed-94f2-400e-97c4-e3fa723424d1 nodeName:}" failed. No retries permitted until 2026-10-30 22:25:22.316313873 +0000 UTC m=+2573.050676888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-maas-api-tls") pod "maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d" (UID: "9060a9ed-94f2-400e-97c4-e3fa723424d1") : secret "maas-api-e2e-praxis-b-37a390-serving-cert" not found
Oct 30 22:25:21.816751 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.816729    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9060a9ed-94f2-400e-97c4-e3fa723424d1-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:21.818931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.818901    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-metrics-certs\") pod \"maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:21.874737 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:21.874698    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hfg5\" (UniqueName: \"kubernetes.io/projected/9060a9ed-94f2-400e-97c4-e3fa723424d1-kube-api-access-9hfg5\") pod \"maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:22.321863 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:22.321804    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-maas-api-tls\") pod \"maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:22.324610 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:22.324583    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-maas-api-tls\") pod \"maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:22.592144 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:22.592110    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:22.977600 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:22.977569    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"]
Oct 30 22:25:22.979226 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:25:22.979201    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9060a9ed_94f2_400e_97c4_e3fa723424d1.slice/crio-bc526846bec3f93bb8673828cfd2288db809e3aa9c41089f24a750dd9ea55d91 WatchSource:0}: Error finding container bc526846bec3f93bb8673828cfd2288db809e3aa9c41089f24a750dd9ea55d91: Status 404 returned error can't find the container with id bc526846bec3f93bb8673828cfd2288db809e3aa9c41089f24a750dd9ea55d91
Oct 30 22:25:23.360555 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:23.360513    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d" event={"ID":"9060a9ed-94f2-400e-97c4-e3fa723424d1","Type":"ContainerStarted","Data":"bc526846bec3f93bb8673828cfd2288db809e3aa9c41089f24a750dd9ea55d91"}
Oct 30 22:25:24.347642 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:24.347603    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"]
Oct 30 22:25:24.348163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:24.347860    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h" podUID="8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" containerName="istio-proxy" containerID="cri-o://7f5f959d1db7353803057a5cd6ea52c34476ed12451c8dbe67523e16dc70f42b" gracePeriod=30
Oct 30 22:25:24.368505 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:24.368472    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d" event={"ID":"9060a9ed-94f2-400e-97c4-e3fa723424d1","Type":"ContainerStarted","Data":"029e2dd5a2de069591c59831d1e172f4f533bd287f411b81be0d54f9ee355c13"}
Oct 30 22:25:24.368672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:24.368526    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:24.591803 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:24.591745    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d" podStartSLOduration=2.804104518 podStartE2EDuration="3.591730077s" totalImagesPullingTime="787.625559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:24.589452011 +0000 UTC m=+2575.323815043" watchObservedRunningTime="2026-10-30 22:25:24.591730077 +0000 UTC m=+2575.326093107"
Oct 30 22:25:26.673051 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.673013    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"]
Oct 30 22:25:26.677289 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.677258    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.730294 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.730245    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-a853fe-openshift-default-dockercfg-r7pcc\""
Oct 30 22:25:26.778498 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.778461    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgmfc\" (UniqueName: \"kubernetes.io/projected/6a77b4ea-e62c-4f91-9264-d037dbe9210e-kube-api-access-mgmfc\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.778669 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.778525    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-credential-socket\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.778752 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.778685    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-workload-socket\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.778752 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.778737    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-envoy\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.778855 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.778817    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istiod-ca-cert\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.778909 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.778852    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-podinfo\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.778909 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.778890    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-data\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.779050 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.779026    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-token\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.779251 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.779115    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-workload-certs\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.785221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.785191    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"]
Oct 30 22:25:26.880690 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.880652    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgmfc\" (UniqueName: \"kubernetes.io/projected/6a77b4ea-e62c-4f91-9264-d037dbe9210e-kube-api-access-mgmfc\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.881004 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.880982    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-credential-socket\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.881210 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.881180    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-workload-socket\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.881373 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.881355    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-envoy\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.881564 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.881536    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istiod-ca-cert\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.883370 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.881589    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-podinfo\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.883370 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.881601    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-credential-socket\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.883370 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.881632    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-data\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.883370 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.881680    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-token\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.883370 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.881773    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-workload-certs\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.883370 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.882187    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-workload-certs\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.889177 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.884615    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istiod-ca-cert\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.889177 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.884670    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-workload-socket\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.889177 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.884730    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-data\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.889177 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.886056    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-podinfo\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.889177 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.888332    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-envoy\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.939089 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.939003    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgmfc\" (UniqueName: \"kubernetes.io/projected/6a77b4ea-e62c-4f91-9264-d037dbe9210e-kube-api-access-mgmfc\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.939338 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.939320    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-token\") pod \"e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") " pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:26.991927 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:26.991891    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:27.424918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:27.424891    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"]
Oct 30 22:25:27.425850 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:25:27.425826    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a77b4ea_e62c_4f91_9264_d037dbe9210e.slice/crio-9aa8cecee2f1f133db9759df7c37750bf71fcca39c88ededd0e93cb89d68ca07 WatchSource:0}: Error finding container 9aa8cecee2f1f133db9759df7c37750bf71fcca39c88ededd0e93cb89d68ca07: Status 404 returned error can't find the container with id 9aa8cecee2f1f133db9759df7c37750bf71fcca39c88ededd0e93cb89d68ca07
Oct 30 22:25:27.428441 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:27.428409    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:25:27.428523 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:27.428476    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:25:27.428523 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:27.428505    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:25:28.203145 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:25:28.203093    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:25:28.395980 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:28.395949    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc" event={"ID":"6a77b4ea-e62c-4f91-9264-d037dbe9210e","Type":"ContainerStarted","Data":"77d73cb21519996a3c577fe7a525a04b5d5cd26be8d2645ce40d781c3370170e"}
Oct 30 22:25:28.396183 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:28.395998    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc" event={"ID":"6a77b4ea-e62c-4f91-9264-d037dbe9210e","Type":"ContainerStarted","Data":"9aa8cecee2f1f133db9759df7c37750bf71fcca39c88ededd0e93cb89d68ca07"}
Oct 30 22:25:28.516603 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:28.516492    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc" podStartSLOduration=2.516478541 podStartE2EDuration="2.516478541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:28.513248971 +0000 UTC m=+2579.247612003" watchObservedRunningTime="2026-10-30 22:25:28.516478541 +0000 UTC m=+2579.250841571"
Oct 30 22:25:28.992493 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:28.992451    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:28.998201 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:28.998169    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:29.402773 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.402744    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:29.403699 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.403657    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:25:29.411087 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:25:29.411060    4553 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0ae1bf_9305_4f6f_a581_475f9a8cc7e9.slice/crio-7f5f959d1db7353803057a5cd6ea52c34476ed12451c8dbe67523e16dc70f42b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0ae1bf_9305_4f6f_a581_475f9a8cc7e9.slice/crio-conmon-7f5f959d1db7353803057a5cd6ea52c34476ed12451c8dbe67523e16dc70f42b.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:25:29.651116 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.651090    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:25:29.713016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.712980    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-kube-api-access-zzj7t\" (UniqueName: \"kubernetes.io/projected/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-kube-api-access-zzj7t\") pod \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") "
Oct 30 22:25:29.713266 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.713042    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-workload-certs\") pod \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") "
Oct 30 22:25:29.713266 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.713072    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-data\") pod \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") "
Oct 30 22:25:29.713266 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.713102    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-credential-socket\") pod \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") "
Oct 30 22:25:29.713266 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.713117    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-token\" (UniqueName: \"kubernetes.io/projected/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-token\") pod \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") "
Oct 30 22:25:29.713266 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.713264    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-podinfo\") pod \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") "
Oct 30 22:25:29.713574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.713336    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-envoy\") pod \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") "
Oct 30 22:25:29.713574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.713404    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-workload-socket\") pod \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") "
Oct 30 22:25:29.713574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.713423    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-workload-certs" pod "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" (UID: "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:29.713574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.713437    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-data" pod "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" (UID: "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:29.713574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.713432    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istiod-ca-cert\") pod \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\" (UID: \"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9\") "
Oct 30 22:25:29.713574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.713450    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-credential-socket" pod "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" (UID: "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:29.713854 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.713694    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-workload-socket" pod "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" (UID: "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:29.713854 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.713770    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istiod-ca-cert" pod "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" (UID: "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:29.714160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.714043    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:29.714281 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.714195    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:29.714281 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.714263    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:29.714281 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.714277    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:29.714465 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.714290    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:29.715693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.715672    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-kube-api-access-zzj7t" pod "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" (UID: "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9"). InnerVolumeSpecName "kube-api-access-zzj7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:29.715794 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.715704    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-envoy" pod "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" (UID: "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:25:29.715855 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.715808    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-podinfo" pod "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" (UID: "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:25:29.715855 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.715822    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-token" pod "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" (UID: "8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:29.815414 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.815378    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:29.815414 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.815410    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:29.815414 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.815420    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzj7t\" (UniqueName: \"kubernetes.io/projected/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-kube-api-access-zzj7t\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:29.815649 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:29.815429    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:30.366020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:30.365986    4553 scope.go:122] "RemoveContainer" containerID="4706cadba9d79cc6d88d1638cf5540529cefb8eb496a68b4ef0259ea720ed08d"
Oct 30 22:25:30.377016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:30.376989    4553 scope.go:122] "RemoveContainer" containerID="7f5f959d1db7353803057a5cd6ea52c34476ed12451c8dbe67523e16dc70f42b"
Oct 30 22:25:30.380840 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:30.380819    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:25:30.408993 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:30.408970    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"
Oct 30 22:25:30.409342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:30.409006    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h" event={"ID":"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9","Type":"ContainerDied","Data":"7f5f959d1db7353803057a5cd6ea52c34476ed12451c8dbe67523e16dc70f42b"}
Oct 30 22:25:30.409342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:30.409058    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h" event={"ID":"8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9","Type":"ContainerDied","Data":"26674ba240ceedc039d9f76888df010a406ca54d2d40f1069af2bc22f88753c9"}
Oct 30 22:25:30.615205 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:30.615167    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"]
Oct 30 22:25:30.617887 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:30.617828    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-96567a-openshift-default-564c67b55c-79r8h"]
Oct 30 22:25:31.157532 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.157493    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"]
Oct 30 22:25:31.159770 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.159733    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq" podUID="4276289b-13bb-4d36-8032-0d12341413b4" containerName="maas-api" containerID="cri-o://04ed7525a89e1f736540bd064d76d8498d299edee879caf3dace0e3f51aa79cd" gracePeriod=30
Oct 30 22:25:31.398451 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.398412    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" path="/var/lib/kubelet/pods/8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9/volumes"
Oct 30 22:25:31.417805 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.417722    4553 generic.go:356] "Generic (PLEG): container finished" podID="4276289b-13bb-4d36-8032-0d12341413b4" containerID="04ed7525a89e1f736540bd064d76d8498d299edee879caf3dace0e3f51aa79cd" exitCode=0
Oct 30 22:25:31.418242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.417798    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq" event={"ID":"4276289b-13bb-4d36-8032-0d12341413b4","Type":"ContainerDied","Data":"04ed7525a89e1f736540bd064d76d8498d299edee879caf3dace0e3f51aa79cd"}
Oct 30 22:25:31.446824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.446800    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:25:31.550787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.550753    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4276289b-13bb-4d36-8032-0d12341413b4-kube-api-access-2t7g8\" (UniqueName: \"kubernetes.io/projected/4276289b-13bb-4d36-8032-0d12341413b4-kube-api-access-2t7g8\") pod \"4276289b-13bb-4d36-8032-0d12341413b4\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") "
Oct 30 22:25:31.551003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.550899    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-metrics-certs\") pod \"4276289b-13bb-4d36-8032-0d12341413b4\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") "
Oct 30 22:25:31.551110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.551095    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4276289b-13bb-4d36-8032-0d12341413b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4276289b-13bb-4d36-8032-0d12341413b4-openshift-service-ca\") pod \"4276289b-13bb-4d36-8032-0d12341413b4\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") "
Oct 30 22:25:31.551951 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.551915    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4276289b-13bb-4d36-8032-0d12341413b4-openshift-service-ca" pod "4276289b-13bb-4d36-8032-0d12341413b4" (UID: "4276289b-13bb-4d36-8032-0d12341413b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:31.552122 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.552015    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-maas-api-tls\") pod \"4276289b-13bb-4d36-8032-0d12341413b4\" (UID: \"4276289b-13bb-4d36-8032-0d12341413b4\") "
Oct 30 22:25:31.553391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.553355    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4276289b-13bb-4d36-8032-0d12341413b4-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.554093 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.554061    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-metrics-certs" pod "4276289b-13bb-4d36-8032-0d12341413b4" (UID: "4276289b-13bb-4d36-8032-0d12341413b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:31.554693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.554662    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4276289b-13bb-4d36-8032-0d12341413b4-kube-api-access-2t7g8" pod "4276289b-13bb-4d36-8032-0d12341413b4" (UID: "4276289b-13bb-4d36-8032-0d12341413b4"). InnerVolumeSpecName "kube-api-access-2t7g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:31.555100 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.555077    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-maas-api-tls" pod "4276289b-13bb-4d36-8032-0d12341413b4" (UID: "4276289b-13bb-4d36-8032-0d12341413b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:31.652650 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.652616    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"]
Oct 30 22:25:31.652862 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.652824    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb" podUID="268ee85a-c628-46c8-bae8-b92ed0456bb8" containerName="payload-processing" containerID="cri-o://e67352530a43d3f62413e851a4c37e84f66934c3763c2372d24f4ea1169ef22c" gracePeriod=30
Oct 30 22:25:31.655109 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.655079    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.655316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.655112    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t7g8\" (UniqueName: \"kubernetes.io/projected/4276289b-13bb-4d36-8032-0d12341413b4-kube-api-access-2t7g8\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.655316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.655129    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4276289b-13bb-4d36-8032-0d12341413b4-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:31.709813 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.709731    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"]
Oct 30 22:25:31.709973 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.709955    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx" podUID="030bf2d1-9267-4507-86fa-c171a880b693" containerName="payload-pre-processing" containerID="cri-o://73b8c8979b3cc03e6a8d7b9c1f8ebae28dac5f9dc8991c6915ac2625eed84f0e" gracePeriod=30
Oct 30 22:25:31.978469 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:31.978434    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"
Oct 30 22:25:32.018785 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.018758    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"
Oct 30 22:25:32.060149 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.060108    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/268ee85a-c628-46c8-bae8-b92ed0456bb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268ee85a-c628-46c8-bae8-b92ed0456bb8-plugins-config-volume\") pod \"268ee85a-c628-46c8-bae8-b92ed0456bb8\" (UID: \"268ee85a-c628-46c8-bae8-b92ed0456bb8\") "
Oct 30 22:25:32.060332 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.060299    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268ee85a-c628-46c8-bae8-b92ed0456bb8-kube-api-access-54sj5\" (UniqueName: \"kubernetes.io/projected/268ee85a-c628-46c8-bae8-b92ed0456bb8-kube-api-access-54sj5\") pod \"268ee85a-c628-46c8-bae8-b92ed0456bb8\" (UID: \"268ee85a-c628-46c8-bae8-b92ed0456bb8\") "
Oct 30 22:25:32.060476 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.060450    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268ee85a-c628-46c8-bae8-b92ed0456bb8-plugins-config-volume" pod "268ee85a-c628-46c8-bae8-b92ed0456bb8" (UID: "268ee85a-c628-46c8-bae8-b92ed0456bb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:32.060716 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.060700    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268ee85a-c628-46c8-bae8-b92ed0456bb8-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:32.062560 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.062539    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268ee85a-c628-46c8-bae8-b92ed0456bb8-kube-api-access-54sj5" pod "268ee85a-c628-46c8-bae8-b92ed0456bb8" (UID: "268ee85a-c628-46c8-bae8-b92ed0456bb8"). InnerVolumeSpecName "kube-api-access-54sj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:32.161420 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.161387    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/030bf2d1-9267-4507-86fa-c171a880b693-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030bf2d1-9267-4507-86fa-c171a880b693-plugins-config-volume\") pod \"030bf2d1-9267-4507-86fa-c171a880b693\" (UID: \"030bf2d1-9267-4507-86fa-c171a880b693\") "
Oct 30 22:25:32.161582 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.161493    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/030bf2d1-9267-4507-86fa-c171a880b693-kube-api-access-4m8dw\" (UniqueName: \"kubernetes.io/projected/030bf2d1-9267-4507-86fa-c171a880b693-kube-api-access-4m8dw\") pod \"030bf2d1-9267-4507-86fa-c171a880b693\" (UID: \"030bf2d1-9267-4507-86fa-c171a880b693\") "
Oct 30 22:25:32.161818 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.161794    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030bf2d1-9267-4507-86fa-c171a880b693-plugins-config-volume" pod "030bf2d1-9267-4507-86fa-c171a880b693" (UID: "030bf2d1-9267-4507-86fa-c171a880b693"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:32.161944 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.161929    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54sj5\" (UniqueName: \"kubernetes.io/projected/268ee85a-c628-46c8-bae8-b92ed0456bb8-kube-api-access-54sj5\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:32.161987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.161952    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/030bf2d1-9267-4507-86fa-c171a880b693-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:32.163863 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.163840    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030bf2d1-9267-4507-86fa-c171a880b693-kube-api-access-4m8dw" pod "030bf2d1-9267-4507-86fa-c171a880b693" (UID: "030bf2d1-9267-4507-86fa-c171a880b693"). InnerVolumeSpecName "kube-api-access-4m8dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:32.263814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.263728    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m8dw\" (UniqueName: \"kubernetes.io/projected/030bf2d1-9267-4507-86fa-c171a880b693-kube-api-access-4m8dw\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:25:32.424537 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.424504    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"
Oct 30 22:25:32.424537 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.424522    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq" event={"ID":"4276289b-13bb-4d36-8032-0d12341413b4","Type":"ContainerDied","Data":"83fc50bafc8c7e8cda919a875fe96eeae81e89adc7226a9d6709e3664129946f"}
Oct 30 22:25:32.425058 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.424586    4553 scope.go:122] "RemoveContainer" containerID="04ed7525a89e1f736540bd064d76d8498d299edee879caf3dace0e3f51aa79cd"
Oct 30 22:25:32.426023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.425923    4553 generic.go:356] "Generic (PLEG): container finished" podID="030bf2d1-9267-4507-86fa-c171a880b693" containerID="73b8c8979b3cc03e6a8d7b9c1f8ebae28dac5f9dc8991c6915ac2625eed84f0e" exitCode=0
Oct 30 22:25:32.426023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.425976    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"
Oct 30 22:25:32.426023 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.426005    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx" event={"ID":"030bf2d1-9267-4507-86fa-c171a880b693","Type":"ContainerDied","Data":"73b8c8979b3cc03e6a8d7b9c1f8ebae28dac5f9dc8991c6915ac2625eed84f0e"}
Oct 30 22:25:32.426233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.426061    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx" event={"ID":"030bf2d1-9267-4507-86fa-c171a880b693","Type":"ContainerDied","Data":"9622ecbde8485b963ebe36b336b675bb6e4ce8b76b104c5a09573477e4ca61b9"}
Oct 30 22:25:32.427692 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.427667    4553 generic.go:356] "Generic (PLEG): container finished" podID="268ee85a-c628-46c8-bae8-b92ed0456bb8" containerID="e67352530a43d3f62413e851a4c37e84f66934c3763c2372d24f4ea1169ef22c" exitCode=0
Oct 30 22:25:32.427834 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.427711    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"
Oct 30 22:25:32.427834 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.427799    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb" event={"ID":"268ee85a-c628-46c8-bae8-b92ed0456bb8","Type":"ContainerDied","Data":"e67352530a43d3f62413e851a4c37e84f66934c3763c2372d24f4ea1169ef22c"}
Oct 30 22:25:32.427946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.427859    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb" event={"ID":"268ee85a-c628-46c8-bae8-b92ed0456bb8","Type":"ContainerDied","Data":"75aeda6a0800274911e593dd917c7cb030c58275501ac40a618ba8218ff3b4ff"}
Oct 30 22:25:32.436859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.436841    4553 scope.go:122] "RemoveContainer" containerID="73b8c8979b3cc03e6a8d7b9c1f8ebae28dac5f9dc8991c6915ac2625eed84f0e"
Oct 30 22:25:32.446284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.446263    4553 scope.go:122] "RemoveContainer" containerID="73b8c8979b3cc03e6a8d7b9c1f8ebae28dac5f9dc8991c6915ac2625eed84f0e"
Oct 30 22:25:32.446592 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:25:32.446576    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b8c8979b3cc03e6a8d7b9c1f8ebae28dac5f9dc8991c6915ac2625eed84f0e\": container with ID starting with 73b8c8979b3cc03e6a8d7b9c1f8ebae28dac5f9dc8991c6915ac2625eed84f0e not found: ID does not exist" containerID="73b8c8979b3cc03e6a8d7b9c1f8ebae28dac5f9dc8991c6915ac2625eed84f0e"
Oct 30 22:25:32.446644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.446602    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b8c8979b3cc03e6a8d7b9c1f8ebae28dac5f9dc8991c6915ac2625eed84f0e"} err="failed to get container status \"73b8c8979b3cc03e6a8d7b9c1f8ebae28dac5f9dc8991c6915ac2625eed84f0e\": rpc error: code = NotFound desc = could not find container \"73b8c8979b3cc03e6a8d7b9c1f8ebae28dac5f9dc8991c6915ac2625eed84f0e\": container with ID starting with 73b8c8979b3cc03e6a8d7b9c1f8ebae28dac5f9dc8991c6915ac2625eed84f0e not found: ID does not exist"
Oct 30 22:25:32.446644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.446622    4553 scope.go:122] "RemoveContainer" containerID="e67352530a43d3f62413e851a4c37e84f66934c3763c2372d24f4ea1169ef22c"
Oct 30 22:25:32.455781 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.455761    4553 scope.go:122] "RemoveContainer" containerID="e67352530a43d3f62413e851a4c37e84f66934c3763c2372d24f4ea1169ef22c"
Oct 30 22:25:32.456166 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:25:32.456118    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e67352530a43d3f62413e851a4c37e84f66934c3763c2372d24f4ea1169ef22c\": container with ID starting with e67352530a43d3f62413e851a4c37e84f66934c3763c2372d24f4ea1169ef22c not found: ID does not exist" containerID="e67352530a43d3f62413e851a4c37e84f66934c3763c2372d24f4ea1169ef22c"
Oct 30 22:25:32.456268 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.456170    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e67352530a43d3f62413e851a4c37e84f66934c3763c2372d24f4ea1169ef22c"} err="failed to get container status \"e67352530a43d3f62413e851a4c37e84f66934c3763c2372d24f4ea1169ef22c\": rpc error: code = NotFound desc = could not find container \"e67352530a43d3f62413e851a4c37e84f66934c3763c2372d24f4ea1169ef22c\": container with ID starting with e67352530a43d3f62413e851a4c37e84f66934c3763c2372d24f4ea1169ef22c not found: ID does not exist"
Oct 30 22:25:32.591014 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.590980    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"]
Oct 30 22:25:32.594958 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.594924    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b1e0a-74675b7d96-jzfqb"]
Oct 30 22:25:32.705158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.705115    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"]
Oct 30 22:25:32.710570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.710528    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8b1e0a-5cb478fd86-lc7kq"]
Oct 30 22:25:32.821460 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.821420    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"]
Oct 30 22:25:32.825300 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:32.825273    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-8b1e0a-5bd7446855-wgrbx"]
Oct 30 22:25:33.394513 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:33.394477    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="030bf2d1-9267-4507-86fa-c171a880b693" path="/var/lib/kubelet/pods/030bf2d1-9267-4507-86fa-c171a880b693/volumes"
Oct 30 22:25:33.394900 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:33.394887    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="268ee85a-c628-46c8-bae8-b92ed0456bb8" path="/var/lib/kubelet/pods/268ee85a-c628-46c8-bae8-b92ed0456bb8/volumes"
Oct 30 22:25:33.395300 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:33.395286    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4276289b-13bb-4d36-8032-0d12341413b4" path="/var/lib/kubelet/pods/4276289b-13bb-4d36-8032-0d12341413b4/volumes"
Oct 30 22:25:37.867644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.867610    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"]
Oct 30 22:25:37.868227 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.868210    4553 cpu_manager.go:414] "containerMap: removing container" podUID="8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" containerName="istio-proxy"
Oct 30 22:25:37.868286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.868229    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" containerName="istio-proxy"
Oct 30 22:25:37.868286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.868244    4553 cpu_manager.go:414] "containerMap: removing container" podUID="030bf2d1-9267-4507-86fa-c171a880b693" containerName="payload-pre-processing"
Oct 30 22:25:37.868286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.868249    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="030bf2d1-9267-4507-86fa-c171a880b693" containerName="payload-pre-processing"
Oct 30 22:25:37.868286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.868262    4553 cpu_manager.go:414] "containerMap: removing container" podUID="4276289b-13bb-4d36-8032-0d12341413b4" containerName="maas-api"
Oct 30 22:25:37.868286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.868270    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4276289b-13bb-4d36-8032-0d12341413b4" containerName="maas-api"
Oct 30 22:25:37.868286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.868284    4553 cpu_manager.go:414] "containerMap: removing container" podUID="268ee85a-c628-46c8-bae8-b92ed0456bb8" containerName="payload-processing"
Oct 30 22:25:37.868286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.868288    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="268ee85a-c628-46c8-bae8-b92ed0456bb8" containerName="payload-processing"
Oct 30 22:25:37.868508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.868310    4553 cpu_manager.go:414] "containerMap: removing container" podUID="8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" containerName="istio-proxy"
Oct 30 22:25:37.868508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.868315    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" containerName="istio-proxy"
Oct 30 22:25:37.868508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.868402    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" containerName="istio-proxy"
Oct 30 22:25:37.868508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.868414    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0ae1bf-9305-4f6f-a581-475f9a8cc7e9" containerName="istio-proxy"
Oct 30 22:25:37.868508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.868427    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="030bf2d1-9267-4507-86fa-c171a880b693" containerName="payload-pre-processing"
Oct 30 22:25:37.868508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.868454    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="4276289b-13bb-4d36-8032-0d12341413b4" containerName="maas-api"
Oct 30 22:25:37.868508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.868461    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="268ee85a-c628-46c8-bae8-b92ed0456bb8" containerName="payload-processing"
Oct 30 22:25:37.873570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.873551    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:37.942162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.941300    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-5b1e39-openshift-default-dockercfg-z569k\""
Oct 30 22:25:37.994981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:37.994942    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"]
Oct 30 22:25:38.041393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.041360    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-data\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.041393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.041397    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t8vr\" (UniqueName: \"kubernetes.io/projected/c02574d3-4aa9-41f0-91c2-759b5d06654e-kube-api-access-4t8vr\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.041693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.041427    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-workload-certs\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.041693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.041452    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-workload-socket\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.041693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.041503    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-token\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.041693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.041575    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02574d3-4aa9-41f0-91c2-759b5d06654e-istiod-ca-cert\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.041693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.041656    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-envoy\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.041693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.041689    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-podinfo\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.042011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.041770    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-credential-socket\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.143623 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.143524    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02574d3-4aa9-41f0-91c2-759b5d06654e-istiod-ca-cert\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.143808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.143627    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-envoy\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.143808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.143659    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-podinfo\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.143808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.143734    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-credential-socket\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.143808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.143797    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-data\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.144040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.143831    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8vr\" (UniqueName: \"kubernetes.io/projected/c02574d3-4aa9-41f0-91c2-759b5d06654e-kube-api-access-4t8vr\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.144040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.143862    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-workload-certs\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.144040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.143889    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-workload-socket\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.144040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.143947    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-token\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.144324 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.144204    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-credential-socket\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.144324 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.144262    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-data\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.144491 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.144471    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-workload-certs\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.144561 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.144490    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-workload-socket\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.144643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.144620    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02574d3-4aa9-41f0-91c2-759b5d06654e-istiod-ca-cert\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.146340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.146317    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-envoy\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.146603 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.146585    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-podinfo\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.202952 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.202912    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t8vr\" (UniqueName: \"kubernetes.io/projected/c02574d3-4aa9-41f0-91c2-759b5d06654e-kube-api-access-4t8vr\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.203391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.203367    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-token\") pod \"e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") " pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.487076 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.486978    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:38.917463 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.917308    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"]
Oct 30 22:25:38.924367 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.924332    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:25:38.924470 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.924405    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:25:38.924470 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:38.924434    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:25:39.475198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:39.475164    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8" event={"ID":"c02574d3-4aa9-41f0-91c2-759b5d06654e","Type":"ContainerStarted","Data":"51ce6e237d8279cf53a6872a6d90773d0eb5b9055f1b1c9805452dfaf6614eec"}
Oct 30 22:25:39.475397 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:39.475219    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8" event={"ID":"c02574d3-4aa9-41f0-91c2-759b5d06654e","Type":"ContainerStarted","Data":"e28c3185a787de440b304bf6f248d811678c150d8a3c645b3042504944c03342"}
Oct 30 22:25:39.487757 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:39.487731    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:39.493226 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:39.493202    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:39.598652 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:39.598599    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8" podStartSLOduration=2.598585262 podStartE2EDuration="2.598585262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:39.595247808 +0000 UTC m=+2590.329610840" watchObservedRunningTime="2026-10-30 22:25:39.598585262 +0000 UTC m=+2590.332948292"
Oct 30 22:25:40.482277 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.482244    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:40.483480 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.483459    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:25:40.591028 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.590994    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"]
Oct 30 22:25:40.595231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.595207    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.648262 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.648223    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-90e0d5-openshift-default-dockercfg-mfg6n\""
Oct 30 22:25:40.678886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.678855    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"]
Oct 30 22:25:40.776693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.776592    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.776693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.776630    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.776926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.776712    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wcln\" (UniqueName: \"kubernetes.io/projected/71ab5f78-088f-4354-b901-760d0b228038-kube-api-access-2wcln\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.776926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.776784    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.776926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.776886    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ab5f78-088f-4354-b901-760d0b228038-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.777069 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.776949    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ab5f78-088f-4354-b901-760d0b228038-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.777069 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.777015    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ab5f78-088f-4354-b901-760d0b228038-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.777069 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.777041    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.777248 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.777223    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.877933 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.877903    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.878125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.877975    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.878125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.878007    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.878125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.878029    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wcln\" (UniqueName: \"kubernetes.io/projected/71ab5f78-088f-4354-b901-760d0b228038-kube-api-access-2wcln\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.878342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.878217    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.878342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.878328    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ab5f78-088f-4354-b901-760d0b228038-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.878453 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.878374    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.878453 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.878417    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ab5f78-088f-4354-b901-760d0b228038-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.878453 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.878430    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.878619 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.878459    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.878619 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.878463    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ab5f78-088f-4354-b901-760d0b228038-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.878619 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.878528    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.878619 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.878570    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.879020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.878998    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ab5f78-088f-4354-b901-760d0b228038-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.880942 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.880916    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.881176 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.881155    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ab5f78-088f-4354-b901-760d0b228038-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.935768 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.935734    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ab5f78-088f-4354-b901-760d0b228038-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:40.935930 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:40.935843    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wcln\" (UniqueName: \"kubernetes.io/projected/71ab5f78-088f-4354-b901-760d0b228038-kube-api-access-2wcln\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:41.209528 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:41.209499    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:41.645276 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:41.645249    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"]
Oct 30 22:25:41.646452 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:25:41.646417    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ab5f78_088f_4354_b901_760d0b228038.slice/crio-d499238aeff8c609dfd98c5372a8eb3afbb70b65b60a725cafe99a16b60f5db1 WatchSource:0}: Error finding container d499238aeff8c609dfd98c5372a8eb3afbb70b65b60a725cafe99a16b60f5db1: Status 404 returned error can't find the container with id d499238aeff8c609dfd98c5372a8eb3afbb70b65b60a725cafe99a16b60f5db1
Oct 30 22:25:41.649399 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:41.649367    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:25:41.649475 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:41.649439    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:25:41.649475 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:41.649468    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:25:42.496432 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:42.496397    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb" event={"ID":"71ab5f78-088f-4354-b901-760d0b228038","Type":"ContainerStarted","Data":"d5a86e717dfd0f1f47b4dd422bef5f0b3b3f714d3d27e84c51f804520f19ad7c"}
Oct 30 22:25:42.496633 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:42.496461    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb" event={"ID":"71ab5f78-088f-4354-b901-760d0b228038","Type":"ContainerStarted","Data":"d499238aeff8c609dfd98c5372a8eb3afbb70b65b60a725cafe99a16b60f5db1"}
Oct 30 22:25:42.667250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:42.667184    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb" podStartSLOduration=2.667168224 podStartE2EDuration="2.667168224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:42.665484228 +0000 UTC m=+2593.399847291" watchObservedRunningTime="2026-10-30 22:25:42.667168224 +0000 UTC m=+2593.401531248"
Oct 30 22:25:43.213158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:43.212495    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:43.232162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:43.216120    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:25:43.232162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:43.216201    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb" podUID="71ab5f78-088f-4354-b901-760d0b228038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 22:25:44.227163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:44.222674    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:25:44.227163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:44.222757    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb" podUID="71ab5f78-088f-4354-b901-760d0b228038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 22:25:45.219343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:45.219294    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:25:45.220418 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:45.220377    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb" podUID="71ab5f78-088f-4354-b901-760d0b228038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 22:25:46.233164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:46.229623    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:25:46.233164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:46.229686    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb" podUID="71ab5f78-088f-4354-b901-760d0b228038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 22:25:47.219161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:47.217633    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:25:47.219161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:47.217728    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb" podUID="71ab5f78-088f-4354-b901-760d0b228038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 22:25:48.209918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:48.209884    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 22:25:48.210354 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:48.209953    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb" podUID="71ab5f78-088f-4354-b901-760d0b228038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 22:25:49.214942 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:49.214908    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:49.215379 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:49.215174    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:49.215941 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:49.215925    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:25:55.150759 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.150721    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"]
Oct 30 22:25:55.154844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.154825    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"
Oct 30 22:25:55.191175 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.191121    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83795218-018c-486a-b209-150f6c424d73-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx\" (UID: \"83795218-018c-486a-b209-150f6c424d73\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"
Oct 30 22:25:55.191353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.191202    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hgl\" (UniqueName: \"kubernetes.io/projected/83795218-018c-486a-b209-150f6c424d73-kube-api-access-69hgl\") pod \"payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx\" (UID: \"83795218-018c-486a-b209-150f6c424d73\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"
Oct 30 22:25:55.230772 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.230732    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"]
Oct 30 22:25:55.261643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.261606    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-a853fe-dockercfg-8vjgb\""
Oct 30 22:25:55.280170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.273280    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-a853fe\""
Oct 30 22:25:55.293495 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.293452    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83795218-018c-486a-b209-150f6c424d73-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx\" (UID: \"83795218-018c-486a-b209-150f6c424d73\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"
Oct 30 22:25:55.293695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.293535    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hgl\" (UniqueName: \"kubernetes.io/projected/83795218-018c-486a-b209-150f6c424d73-kube-api-access-69hgl\") pod \"payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx\" (UID: \"83795218-018c-486a-b209-150f6c424d73\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"
Oct 30 22:25:55.294539 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.294465    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83795218-018c-486a-b209-150f6c424d73-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx\" (UID: \"83795218-018c-486a-b209-150f6c424d73\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"
Oct 30 22:25:55.328497 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.328442    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"]
Oct 30 22:25:55.338539 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.338508    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"
Oct 30 22:25:55.352572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.352542    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hgl\" (UniqueName: \"kubernetes.io/projected/83795218-018c-486a-b209-150f6c424d73-kube-api-access-69hgl\") pod \"payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx\" (UID: \"83795218-018c-486a-b209-150f6c424d73\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"
Oct 30 22:25:55.360536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.360509    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"]
Oct 30 22:25:55.394250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.394211    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86123dfa-c061-4101-a32f-9788677cdb14-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97\" (UID: \"86123dfa-c061-4101-a32f-9788677cdb14\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"
Oct 30 22:25:55.394447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.394291    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrdpx\" (UniqueName: \"kubernetes.io/projected/86123dfa-c061-4101-a32f-9788677cdb14-kube-api-access-xrdpx\") pod \"payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97\" (UID: \"86123dfa-c061-4101-a32f-9788677cdb14\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"
Oct 30 22:25:55.469644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.469557    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"
Oct 30 22:25:55.495724 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.495683    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrdpx\" (UniqueName: \"kubernetes.io/projected/86123dfa-c061-4101-a32f-9788677cdb14-kube-api-access-xrdpx\") pod \"payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97\" (UID: \"86123dfa-c061-4101-a32f-9788677cdb14\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"
Oct 30 22:25:55.496244 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.496048    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86123dfa-c061-4101-a32f-9788677cdb14-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97\" (UID: \"86123dfa-c061-4101-a32f-9788677cdb14\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"
Oct 30 22:25:55.497170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.497108    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86123dfa-c061-4101-a32f-9788677cdb14-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97\" (UID: \"86123dfa-c061-4101-a32f-9788677cdb14\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"
Oct 30 22:25:55.554684 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.554656    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrdpx\" (UniqueName: \"kubernetes.io/projected/86123dfa-c061-4101-a32f-9788677cdb14-kube-api-access-xrdpx\") pod \"payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97\" (UID: \"86123dfa-c061-4101-a32f-9788677cdb14\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"
Oct 30 22:25:55.657189 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.657126    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"
Oct 30 22:25:55.879075 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:55.879042    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"]
Oct 30 22:25:55.881003 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:25:55.880976    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83795218_018c_486a_b209_150f6c424d73.slice/crio-da9648bf2832aade7266446839b0c6b4ed7aebc98310a5cc070ee985906f9df0 WatchSource:0}: Error finding container da9648bf2832aade7266446839b0c6b4ed7aebc98310a5cc070ee985906f9df0: Status 404 returned error can't find the container with id da9648bf2832aade7266446839b0c6b4ed7aebc98310a5cc070ee985906f9df0
Oct 30 22:25:56.065675 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:56.065642    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"]
Oct 30 22:25:56.066550 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:25:56.066517    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86123dfa_c061_4101_a32f_9788677cdb14.slice/crio-6cb35ad31affa268cfec5c58243ed27b169ad11bc7a777e78b84f3045fc9ca97 WatchSource:0}: Error finding container 6cb35ad31affa268cfec5c58243ed27b169ad11bc7a777e78b84f3045fc9ca97: Status 404 returned error can't find the container with id 6cb35ad31affa268cfec5c58243ed27b169ad11bc7a777e78b84f3045fc9ca97
Oct 30 22:25:56.634555 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:56.634513    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx" event={"ID":"83795218-018c-486a-b209-150f6c424d73","Type":"ContainerStarted","Data":"e2db169a00e290e518134deeb143cb8f4ca9d8ebd416ff7d597386a35038d6ca"}
Oct 30 22:25:56.635102 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:56.634586    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx" event={"ID":"83795218-018c-486a-b209-150f6c424d73","Type":"ContainerStarted","Data":"da9648bf2832aade7266446839b0c6b4ed7aebc98310a5cc070ee985906f9df0"}
Oct 30 22:25:56.635102 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:56.634785    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"
Oct 30 22:25:56.636669 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:56.636636    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97" event={"ID":"86123dfa-c061-4101-a32f-9788677cdb14","Type":"ContainerStarted","Data":"456a3b7d5041be643405222909e91e3fa2c4758b5a71eadfff7e4ba679dba2e9"}
Oct 30 22:25:56.636802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:56.636682    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97" event={"ID":"86123dfa-c061-4101-a32f-9788677cdb14","Type":"ContainerStarted","Data":"6cb35ad31affa268cfec5c58243ed27b169ad11bc7a777e78b84f3045fc9ca97"}
Oct 30 22:25:56.636802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:56.636709    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"
Oct 30 22:25:56.754343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:56.754289    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx" podStartSLOduration=1.754273878 podStartE2EDuration="1.754273878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:56.752906173 +0000 UTC m=+2607.487269204" watchObservedRunningTime="2026-10-30 22:25:56.754273878 +0000 UTC m=+2607.488636909"
Oct 30 22:25:56.873403 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:25:56.873348    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97" podStartSLOduration=1.8733318209999998 podStartE2EDuration="1.873331821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:56.868771378 +0000 UTC m=+2607.603134435" watchObservedRunningTime="2026-10-30 22:25:56.873331821 +0000 UTC m=+2607.607694852"
Oct 30 22:25:58.203261 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:25:58.203212    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:26:02.461071 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.461023    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"]
Oct 30 22:26:02.466146 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.466106    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"
Oct 30 22:26:02.521482 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.521452    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"]
Oct 30 22:26:02.592254 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.575457    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-5b1e39\""
Oct 30 22:26:02.592254 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.575610    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-5b1e39-dockercfg-zld96\""
Oct 30 22:26:02.612933 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.612903    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6j92\" (UniqueName: \"kubernetes.io/projected/455fabb6-e529-4bd2-8247-4d49897f3dce-kube-api-access-s6j92\") pod \"payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb\" (UID: \"455fabb6-e529-4bd2-8247-4d49897f3dce\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"
Oct 30 22:26:02.613120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.612952    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455fabb6-e529-4bd2-8247-4d49897f3dce-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb\" (UID: \"455fabb6-e529-4bd2-8247-4d49897f3dce\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"
Oct 30 22:26:02.674888 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.674856    4553 generic.go:356] "Generic (PLEG): container finished" podID="7a2af516-6dad-461a-9674-8951b4231d49" containerID="1293b95ef6bfaf6824859b78f419990577047219114765459a00efb46c2d61f7" exitCode=0
Oct 30 22:26:02.675068 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.674938    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l" event={"ID":"7a2af516-6dad-461a-9674-8951b4231d49","Type":"ContainerDied","Data":"1293b95ef6bfaf6824859b78f419990577047219114765459a00efb46c2d61f7"}
Oct 30 22:26:02.675432 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.675418    4553 scope.go:122] "RemoveContainer" containerID="1293b95ef6bfaf6824859b78f419990577047219114765459a00efb46c2d61f7"
Oct 30 22:26:02.677336 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.677305    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:26:02.677526 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.677394    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:26:02.677526 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.677443    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:26:02.714384 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.714344    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6j92\" (UniqueName: \"kubernetes.io/projected/455fabb6-e529-4bd2-8247-4d49897f3dce-kube-api-access-s6j92\") pod \"payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb\" (UID: \"455fabb6-e529-4bd2-8247-4d49897f3dce\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"
Oct 30 22:26:02.714545 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.714399    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455fabb6-e529-4bd2-8247-4d49897f3dce-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb\" (UID: \"455fabb6-e529-4bd2-8247-4d49897f3dce\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"
Oct 30 22:26:02.715083 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.715063    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455fabb6-e529-4bd2-8247-4d49897f3dce-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb\" (UID: \"455fabb6-e529-4bd2-8247-4d49897f3dce\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"
Oct 30 22:26:02.772091 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.772060    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6j92\" (UniqueName: \"kubernetes.io/projected/455fabb6-e529-4bd2-8247-4d49897f3dce-kube-api-access-s6j92\") pod \"payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb\" (UID: \"455fabb6-e529-4bd2-8247-4d49897f3dce\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"
Oct 30 22:26:02.843507 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:02.843479    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"
Oct 30 22:26:03.231365 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:03.231331    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"]
Oct 30 22:26:03.232698 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:26:03.232671    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455fabb6_e529_4bd2_8247_4d49897f3dce.slice/crio-30070c39949815e57a871077f359112813010587182f0d1c2a98f329af8f39d1 WatchSource:0}: Error finding container 30070c39949815e57a871077f359112813010587182f0d1c2a98f329af8f39d1: Status 404 returned error can't find the container with id 30070c39949815e57a871077f359112813010587182f0d1c2a98f329af8f39d1
Oct 30 22:26:03.683087 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:03.683047    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l" event={"ID":"7a2af516-6dad-461a-9674-8951b4231d49","Type":"ContainerStarted","Data":"cc0e2cc9c2344e134571d4d7790e6ebbf35ebe7cf8c9cd79cb7068a2e8f3bbec"}
Oct 30 22:26:03.684791 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:03.684763    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb" event={"ID":"455fabb6-e529-4bd2-8247-4d49897f3dce","Type":"ContainerStarted","Data":"f49c56dabc118b22d990f0bc7173676bbbbb520be434b436337df0f6a8617d05"}
Oct 30 22:26:03.684913 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:03.684809    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb" event={"ID":"455fabb6-e529-4bd2-8247-4d49897f3dce","Type":"ContainerStarted","Data":"30070c39949815e57a871077f359112813010587182f0d1c2a98f329af8f39d1"}
Oct 30 22:26:03.684913 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:03.684897    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"
Oct 30 22:26:03.930406 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:03.930336    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb" podStartSLOduration=1.930317158 podStartE2EDuration="1.930317158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:03.929831899 +0000 UTC m=+2614.664194943" watchObservedRunningTime="2026-10-30 22:26:03.930317158 +0000 UTC m=+2614.664680190"
Oct 30 22:26:04.613926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:04.613684    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:26:04.637724 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:04.637313    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:26:04.637724 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:04.637396    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l" podUID="7a2af516-6dad-461a-9674-8951b4231d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 22:26:05.633564 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:05.633524    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:26:05.634109 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:05.633588    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l" podUID="7a2af516-6dad-461a-9674-8951b4231d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 22:26:05.933391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:05.933282    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:26:06.615163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:06.613529    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:26:06.615163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:06.613592    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l" podUID="7a2af516-6dad-461a-9674-8951b4231d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 22:26:07.622162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:07.614567    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:26:07.622162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:07.614636    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l" podUID="7a2af516-6dad-461a-9674-8951b4231d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 22:26:07.646071 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:07.646040    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"
Oct 30 22:26:07.646329 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:07.646119    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"
Oct 30 22:26:08.621157 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:08.613591    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:26:08.621157 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:08.613689    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l" podUID="7a2af516-6dad-461a-9674-8951b4231d49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Oct 30 22:26:09.616537 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:09.616507    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:26:09.617681 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:09.617661    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:26:14.694754 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:14.694720    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"
Oct 30 22:26:25.622485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.622444    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"]
Oct 30 22:26:25.626617 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.626597    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.724628 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.724587    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltbks\" (UniqueName: \"kubernetes.io/projected/c75e16cc-0291-4c10-844c-cd8de419f01f-kube-api-access-ltbks\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.724805 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.724722    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.724805 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.724778    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.724927 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.724813    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.724990 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.724948    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.725046 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.724986    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75e16cc-0291-4c10-844c-cd8de419f01f-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.725127 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.725107    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.725220 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.725192    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.725332 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.725225    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.733654 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.733620    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"]
Oct 30 22:26:25.826793 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.826757    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.826982 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.826823    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.826982 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.826878    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.826982 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.826905    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75e16cc-0291-4c10-844c-cd8de419f01f-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.827174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.826985    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.827174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.827033    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.827174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.827073    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.827351 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.827203    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltbks\" (UniqueName: \"kubernetes.io/projected/c75e16cc-0291-4c10-844c-cd8de419f01f-kube-api-access-ltbks\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.827351 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.827308    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.827351 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.827323    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.827763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.827716    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.827902 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.827776    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.827971 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.827901    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.828546 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.828519    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75e16cc-0291-4c10-844c-cd8de419f01f-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.829939 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.829914    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.830442 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.830420    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.884896 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.884826    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.885112 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.885087    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltbks\" (UniqueName: \"kubernetes.io/projected/c75e16cc-0291-4c10-844c-cd8de419f01f-kube-api-access-ltbks\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:25.940758 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:25.940724    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:26.376785 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:26.376746    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"]
Oct 30 22:26:26.378121 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:26:26.378094    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75e16cc_0291_4c10_844c_cd8de419f01f.slice/crio-50519ca02b10ef7e60fd1744240afb3b5a33fa7bb91fc9917cc2ee76a38b87d8 WatchSource:0}: Error finding container 50519ca02b10ef7e60fd1744240afb3b5a33fa7bb91fc9917cc2ee76a38b87d8: Status 404 returned error can't find the container with id 50519ca02b10ef7e60fd1744240afb3b5a33fa7bb91fc9917cc2ee76a38b87d8
Oct 30 22:26:26.380739 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:26.380708    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:26:26.380830 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:26.380776    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:26:26.380830 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:26.380802    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:26:26.866769 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:26.866733    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq" event={"ID":"c75e16cc-0291-4c10-844c-cd8de419f01f","Type":"ContainerStarted","Data":"0b8f38ecd59811302b114a1ed88dbea027da4070411018af6995fa3dd22331e1"}
Oct 30 22:26:26.867247 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:26.866819    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq" event={"ID":"c75e16cc-0291-4c10-844c-cd8de419f01f","Type":"ContainerStarted","Data":"50519ca02b10ef7e60fd1744240afb3b5a33fa7bb91fc9917cc2ee76a38b87d8"}
Oct 30 22:26:26.941109 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:26.941073    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:27.944681 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:27.944645    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:26:27.945222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:27.944703    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq" podUID="c75e16cc-0291-4c10-844c-cd8de419f01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 22:26:28.205050 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:26:28.204966    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:26:28.941845 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:28.941811    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:26:28.942034 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:28.941860    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq" podUID="c75e16cc-0291-4c10-844c-cd8de419f01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 22:26:29.941823 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:29.941794    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:26:29.942257 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:29.941845    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq" podUID="c75e16cc-0291-4c10-844c-cd8de419f01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 22:26:30.942652 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:30.942618    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:26:30.943182 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:30.942690    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq" podUID="c75e16cc-0291-4c10-844c-cd8de419f01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 22:26:31.941737 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:31.941705    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:26:31.941924 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:31.941759    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq" podUID="c75e16cc-0291-4c10-844c-cd8de419f01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 22:26:32.941358 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:32.941320    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:26:32.941868 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:32.941396    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq" podUID="c75e16cc-0291-4c10-844c-cd8de419f01f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused"
Oct 30 22:26:33.946220 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:33.946190    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:33.946621 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:33.946605    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:33.947423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:33.947408    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:26:34.118114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:34.118059    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq" podStartSLOduration=9.1180454 podStartE2EDuration="9.1180454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:27.035475091 +0000 UTC m=+2637.769838121" watchObservedRunningTime="2026-10-30 22:26:34.1180454 +0000 UTC m=+2644.852408431"
Oct 30 22:26:36.117632 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.117596    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"]
Oct 30 22:26:36.118016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.117845    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l" podUID="dcc1f853-d639-4df7-b61f-06232e9a6c42" containerName="payload-processing" containerID="cri-o://19bdf48fed17919188e12baa4ea6fffad2ccba05591ffb9158bbbbd45de67457" gracePeriod=30
Oct 30 22:26:36.187976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.187930    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l" podUID="dcc1f853-d639-4df7-b61f-06232e9a6c42" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.161:9004: connect: connection refused"
Oct 30 22:26:36.228778 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.228736    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"]
Oct 30 22:26:36.229018 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.228988    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf" podUID="256e7ef9-77ab-424a-910f-af05d51522a3" containerName="payload-pre-processing" containerID="cri-o://39fc54eef9e01df2ee34e360a2d3aeed880b1a41b094e499b5ee8d02d97cd352" gracePeriod=30
Oct 30 22:26:36.433871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.433557    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"
Oct 30 22:26:36.506177 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.506152    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"
Oct 30 22:26:36.566889 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.566857    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/256e7ef9-77ab-424a-910f-af05d51522a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256e7ef9-77ab-424a-910f-af05d51522a3-plugins-config-volume\") pod \"256e7ef9-77ab-424a-910f-af05d51522a3\" (UID: \"256e7ef9-77ab-424a-910f-af05d51522a3\") "
Oct 30 22:26:36.567123 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.567031    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256e7ef9-77ab-424a-910f-af05d51522a3-kube-api-access-zkvzz\" (UniqueName: \"kubernetes.io/projected/256e7ef9-77ab-424a-910f-af05d51522a3-kube-api-access-zkvzz\") pod \"256e7ef9-77ab-424a-910f-af05d51522a3\" (UID: \"256e7ef9-77ab-424a-910f-af05d51522a3\") "
Oct 30 22:26:36.567280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.567259    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc1f853-d639-4df7-b61f-06232e9a6c42-kube-api-access-rfnb4\" (UniqueName: \"kubernetes.io/projected/dcc1f853-d639-4df7-b61f-06232e9a6c42-kube-api-access-rfnb4\") pod \"dcc1f853-d639-4df7-b61f-06232e9a6c42\" (UID: \"dcc1f853-d639-4df7-b61f-06232e9a6c42\") "
Oct 30 22:26:36.567355 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.567321    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/256e7ef9-77ab-424a-910f-af05d51522a3-plugins-config-volume" pod "256e7ef9-77ab-424a-910f-af05d51522a3" (UID: "256e7ef9-77ab-424a-910f-af05d51522a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:36.567355 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.567349    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcc1f853-d639-4df7-b61f-06232e9a6c42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcc1f853-d639-4df7-b61f-06232e9a6c42-plugins-config-volume\") pod \"dcc1f853-d639-4df7-b61f-06232e9a6c42\" (UID: \"dcc1f853-d639-4df7-b61f-06232e9a6c42\") "
Oct 30 22:26:36.567744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.567720    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc1f853-d639-4df7-b61f-06232e9a6c42-plugins-config-volume" pod "dcc1f853-d639-4df7-b61f-06232e9a6c42" (UID: "dcc1f853-d639-4df7-b61f-06232e9a6c42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:36.568208 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.568182    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256e7ef9-77ab-424a-910f-af05d51522a3-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:36.568335 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.568209    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcc1f853-d639-4df7-b61f-06232e9a6c42-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:36.569481 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.569447    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256e7ef9-77ab-424a-910f-af05d51522a3-kube-api-access-zkvzz" pod "256e7ef9-77ab-424a-910f-af05d51522a3" (UID: "256e7ef9-77ab-424a-910f-af05d51522a3"). InnerVolumeSpecName "kube-api-access-zkvzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:36.569645 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.569624    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc1f853-d639-4df7-b61f-06232e9a6c42-kube-api-access-rfnb4" pod "dcc1f853-d639-4df7-b61f-06232e9a6c42" (UID: "dcc1f853-d639-4df7-b61f-06232e9a6c42"). InnerVolumeSpecName "kube-api-access-rfnb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:36.706335 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.702667    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfnb4\" (UniqueName: \"kubernetes.io/projected/dcc1f853-d639-4df7-b61f-06232e9a6c42-kube-api-access-rfnb4\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:36.706335 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.706288    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkvzz\" (UniqueName: \"kubernetes.io/projected/256e7ef9-77ab-424a-910f-af05d51522a3-kube-api-access-zkvzz\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:36.933734 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.933695    4553 generic.go:356] "Generic (PLEG): container finished" podID="dcc1f853-d639-4df7-b61f-06232e9a6c42" containerID="19bdf48fed17919188e12baa4ea6fffad2ccba05591ffb9158bbbbd45de67457" exitCode=0
Oct 30 22:26:36.933943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.933765    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"
Oct 30 22:26:36.933943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.933794    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l" event={"ID":"dcc1f853-d639-4df7-b61f-06232e9a6c42","Type":"ContainerDied","Data":"19bdf48fed17919188e12baa4ea6fffad2ccba05591ffb9158bbbbd45de67457"}
Oct 30 22:26:36.933943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.933862    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l" event={"ID":"dcc1f853-d639-4df7-b61f-06232e9a6c42","Type":"ContainerDied","Data":"e3b289dd79f58c85eb0f91b4f686c085260a2a6bdbcf23c636b37238ccda6c66"}
Oct 30 22:26:36.933943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.933894    4553 scope.go:122] "RemoveContainer" containerID="19bdf48fed17919188e12baa4ea6fffad2ccba05591ffb9158bbbbd45de67457"
Oct 30 22:26:36.935405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.935381    4553 generic.go:356] "Generic (PLEG): container finished" podID="256e7ef9-77ab-424a-910f-af05d51522a3" containerID="39fc54eef9e01df2ee34e360a2d3aeed880b1a41b094e499b5ee8d02d97cd352" exitCode=0
Oct 30 22:26:36.935535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.935427    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"
Oct 30 22:26:36.935600 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.935541    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf" event={"ID":"256e7ef9-77ab-424a-910f-af05d51522a3","Type":"ContainerDied","Data":"39fc54eef9e01df2ee34e360a2d3aeed880b1a41b094e499b5ee8d02d97cd352"}
Oct 30 22:26:36.935600 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.935577    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf" event={"ID":"256e7ef9-77ab-424a-910f-af05d51522a3","Type":"ContainerDied","Data":"b96a766d856eb1c94213f3146f96a03b9cf874cf6541a05f4dc8e14ab1d1bcb6"}
Oct 30 22:26:36.946086 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.946068    4553 scope.go:122] "RemoveContainer" containerID="19bdf48fed17919188e12baa4ea6fffad2ccba05591ffb9158bbbbd45de67457"
Oct 30 22:26:36.946452 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:26:36.946422    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19bdf48fed17919188e12baa4ea6fffad2ccba05591ffb9158bbbbd45de67457\": container with ID starting with 19bdf48fed17919188e12baa4ea6fffad2ccba05591ffb9158bbbbd45de67457 not found: ID does not exist" containerID="19bdf48fed17919188e12baa4ea6fffad2ccba05591ffb9158bbbbd45de67457"
Oct 30 22:26:36.946553 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.946457    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19bdf48fed17919188e12baa4ea6fffad2ccba05591ffb9158bbbbd45de67457"} err="failed to get container status \"19bdf48fed17919188e12baa4ea6fffad2ccba05591ffb9158bbbbd45de67457\": rpc error: code = NotFound desc = could not find container \"19bdf48fed17919188e12baa4ea6fffad2ccba05591ffb9158bbbbd45de67457\": container with ID starting with 19bdf48fed17919188e12baa4ea6fffad2ccba05591ffb9158bbbbd45de67457 not found: ID does not exist"
Oct 30 22:26:36.946553 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.946480    4553 scope.go:122] "RemoveContainer" containerID="39fc54eef9e01df2ee34e360a2d3aeed880b1a41b094e499b5ee8d02d97cd352"
Oct 30 22:26:36.956316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.956297    4553 scope.go:122] "RemoveContainer" containerID="39fc54eef9e01df2ee34e360a2d3aeed880b1a41b094e499b5ee8d02d97cd352"
Oct 30 22:26:36.956611 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:26:36.956571    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39fc54eef9e01df2ee34e360a2d3aeed880b1a41b094e499b5ee8d02d97cd352\": container with ID starting with 39fc54eef9e01df2ee34e360a2d3aeed880b1a41b094e499b5ee8d02d97cd352 not found: ID does not exist" containerID="39fc54eef9e01df2ee34e360a2d3aeed880b1a41b094e499b5ee8d02d97cd352"
Oct 30 22:26:36.956668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:36.956603    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39fc54eef9e01df2ee34e360a2d3aeed880b1a41b094e499b5ee8d02d97cd352"} err="failed to get container status \"39fc54eef9e01df2ee34e360a2d3aeed880b1a41b094e499b5ee8d02d97cd352\": rpc error: code = NotFound desc = could not find container \"39fc54eef9e01df2ee34e360a2d3aeed880b1a41b094e499b5ee8d02d97cd352\": container with ID starting with 39fc54eef9e01df2ee34e360a2d3aeed880b1a41b094e499b5ee8d02d97cd352 not found: ID does not exist"
Oct 30 22:26:37.099917 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.099827    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"]
Oct 30 22:26:37.103100 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.103065    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-955887-6b66898f6c-5r4hf"]
Oct 30 22:26:37.170729 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.170696    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"]
Oct 30 22:26:37.171125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.170919    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6" podUID="ef6dce00-fe43-484e-8d2f-3d1e6f480d53" containerName="istio-proxy" containerID="cri-o://c20c412bda9d26913bc233736452221022b6efbbdaefed610e7f4c88652c8a45" gracePeriod=30
Oct 30 22:26:37.217956 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.217875    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"]
Oct 30 22:26:37.220648 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.220624    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-955887-b8d7c5cb4-gqg5l"]
Oct 30 22:26:37.287342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.287312    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"]
Oct 30 22:26:37.287897 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.287884    4553 cpu_manager.go:414] "containerMap: removing container" podUID="256e7ef9-77ab-424a-910f-af05d51522a3" containerName="payload-pre-processing"
Oct 30 22:26:37.287946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.287899    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="256e7ef9-77ab-424a-910f-af05d51522a3" containerName="payload-pre-processing"
Oct 30 22:26:37.287946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.287922    4553 cpu_manager.go:414] "containerMap: removing container" podUID="dcc1f853-d639-4df7-b61f-06232e9a6c42" containerName="payload-processing"
Oct 30 22:26:37.287946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.287928    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc1f853-d639-4df7-b61f-06232e9a6c42" containerName="payload-processing"
Oct 30 22:26:37.288039 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.288013    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc1f853-d639-4df7-b61f-06232e9a6c42" containerName="payload-processing"
Oct 30 22:26:37.288039 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.288028    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="256e7ef9-77ab-424a-910f-af05d51522a3" containerName="payload-pre-processing"
Oct 30 22:26:37.292723 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.292705    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:37.346423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.346390    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-a853fe-serving-cert\""
Oct 30 22:26:37.350607 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.350577    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"]
Oct 30 22:26:37.441160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.415548    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96f7v\" (UniqueName: \"kubernetes.io/projected/15b7811c-45bc-49ff-a3e2-debaedc77a89-kube-api-access-96f7v\") pod \"maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:37.441160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.415624    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-metrics-certs\") pod \"maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:37.441160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.415737    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b7811c-45bc-49ff-a3e2-debaedc77a89-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:37.441160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.415796    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:37.473233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.472338    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="256e7ef9-77ab-424a-910f-af05d51522a3" path="/var/lib/kubelet/pods/256e7ef9-77ab-424a-910f-af05d51522a3/volumes"
Oct 30 22:26:37.473233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.472928    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc1f853-d639-4df7-b61f-06232e9a6c42" path="/var/lib/kubelet/pods/dcc1f853-d639-4df7-b61f-06232e9a6c42/volumes"
Oct 30 22:26:37.517169 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.517111    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96f7v\" (UniqueName: \"kubernetes.io/projected/15b7811c-45bc-49ff-a3e2-debaedc77a89-kube-api-access-96f7v\") pod \"maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:37.517382 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.517199    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-metrics-certs\") pod \"maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:37.517382 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.517296    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b7811c-45bc-49ff-a3e2-debaedc77a89-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:37.517382 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.517351    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:37.517591 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:26:37.517480    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-serving-cert: secret "maas-api-e2e-praxis-a-a853fe-serving-cert" not found
Oct 30 22:26:37.517591 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:26:37.517540    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-maas-api-tls podName:15b7811c-45bc-49ff-a3e2-debaedc77a89 nodeName:}" failed. No retries permitted until 2026-10-30 22:26:38.017519621 +0000 UTC m=+2648.751882636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-maas-api-tls") pod "maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm" (UID: "15b7811c-45bc-49ff-a3e2-debaedc77a89") : secret "maas-api-e2e-praxis-a-a853fe-serving-cert" not found
Oct 30 22:26:37.518432 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.518401    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b7811c-45bc-49ff-a3e2-debaedc77a89-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:37.520505 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.520475    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-metrics-certs\") pod \"maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:37.576746 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:37.576705    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96f7v\" (UniqueName: \"kubernetes.io/projected/15b7811c-45bc-49ff-a3e2-debaedc77a89-kube-api-access-96f7v\") pod \"maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:38.024497 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:38.024449    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:38.027339 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:38.027310    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:38.207074 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:38.207035    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:38.644714 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:38.630721    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"]
Oct 30 22:26:38.973582 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:38.973472    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm" event={"ID":"15b7811c-45bc-49ff-a3e2-debaedc77a89","Type":"ContainerStarted","Data":"ead41af205e4801f6060d5361b0f178c4eec0055544ca6d28db6828df11e3892"}
Oct 30 22:26:39.981082 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:39.981044    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm" event={"ID":"15b7811c-45bc-49ff-a3e2-debaedc77a89","Type":"ContainerStarted","Data":"1015b15cd0fc0b4c82d1ae63dd801c20e728d152fa35fcba662e81c62c1a6c8f"}
Oct 30 22:26:39.981556 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:39.981238    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:40.101256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.101205    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm" podStartSLOduration=2.184994067 podStartE2EDuration="3.101190668s" totalImagesPullingTime="916.196601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:40.099260846 +0000 UTC m=+2650.833623864" watchObservedRunningTime="2026-10-30 22:26:40.101190668 +0000 UTC m=+2650.835553726"
Oct 30 22:26:40.716162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.715645    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"]
Oct 30 22:26:40.721486 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.721447    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.777327 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.777281    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l78tn\" (UniqueName: \"kubernetes.io/projected/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-kube-api-access-l78tn\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.777522 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.777358    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-home\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.777522 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.777393    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-model-cache\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.777522 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.777418    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-tls-certs\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.777522 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.777488    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-dshm\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.777756 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.777531    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-tmp-dir\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.778105 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.778079    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"]
Oct 30 22:26:40.824095 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.824064    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-086575\"/\"kube-root-ca.crt\""
Oct 30 22:26:40.824095 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.824064    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-086575\"/\"default-dockercfg-plpzh\""
Oct 30 22:26:40.824368 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.824146    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-086575\"/\"openshift-service-ca.crt\""
Oct 30 22:26:40.825491 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.825477    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-086575\"/\"praxis-route-086575-kserve-self-signed-certs\""
Oct 30 22:26:40.878886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.878850    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-dshm\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.879110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.878922    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-tmp-dir\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.879110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.879009    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l78tn\" (UniqueName: \"kubernetes.io/projected/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-kube-api-access-l78tn\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.879110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.879100    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-home\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.879340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.879201    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-model-cache\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.879340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.879244    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-tls-certs\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.879458 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.879437    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-tmp-dir\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.879517 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.879485    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-home\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.879575 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.879516    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-model-cache\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.881491 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.881467    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-dshm\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.881837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.881822    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-tls-certs\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:40.937732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:40.937702    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l78tn\" (UniqueName: \"kubernetes.io/projected/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-kube-api-access-l78tn\") pod \"praxis-route-086575-kserve-945fdf7dc-2pdmf\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") " pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:41.038926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:41.038850    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:41.437453 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:41.437418    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"]
Oct 30 22:26:41.438778 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:26:41.438753    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dc9ec8d_22b0_4f4b_aa46_91b76653e0d7.slice/crio-9355e20e88d53e5c50d5a44bb25f2e2fcde10af68f91041b34a6796db413084b WatchSource:0}: Error finding container 9355e20e88d53e5c50d5a44bb25f2e2fcde10af68f91041b34a6796db413084b: Status 404 returned error can't find the container with id 9355e20e88d53e5c50d5a44bb25f2e2fcde10af68f91041b34a6796db413084b
Oct 30 22:26:41.727062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:41.726974    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-pqpw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:26:41.727062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:41.727026    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6" podUID="ef6dce00-fe43-484e-8d2f-3d1e6f480d53" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:26:41.996856 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:41.996764    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf" event={"ID":"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7","Type":"ContainerStarted","Data":"b5cc6b0e6c8e532474a355cbf6307b25ac468f44b45a26e2aa34fcdf8219885a"}
Oct 30 22:26:41.996856 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:41.996814    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf" event={"ID":"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7","Type":"ContainerStarted","Data":"9355e20e88d53e5c50d5a44bb25f2e2fcde10af68f91041b34a6796db413084b"}
Oct 30 22:26:42.039290 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.039249    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:42.116446 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.116391    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf" podStartSLOduration=2.116370859 podStartE2EDuration="2.116370859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:42.114202523 +0000 UTC m=+2652.848565555" watchObservedRunningTime="2026-10-30 22:26:42.116370859 +0000 UTC m=+2652.850733963"
Oct 30 22:26:42.448517 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.448491    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:26:42.499963 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.499926    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-kube-api-access-bstql\" (UniqueName: \"kubernetes.io/projected/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-kube-api-access-bstql\") pod \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") "
Oct 30 22:26:42.500214 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.499980    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istiod-ca-cert\") pod \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") "
Oct 30 22:26:42.500214 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.500098    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-workload-socket\") pod \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") "
Oct 30 22:26:42.500214 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.500194    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-podinfo\") pod \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") "
Oct 30 22:26:42.500352 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.500228    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-credential-socket\") pod \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") "
Oct 30 22:26:42.500352 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.500323    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-workload-certs\") pod \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") "
Oct 30 22:26:42.500510 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.500437    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-envoy\") pod \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") "
Oct 30 22:26:42.500510 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.500440    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-workload-socket" pod "ef6dce00-fe43-484e-8d2f-3d1e6f480d53" (UID: "ef6dce00-fe43-484e-8d2f-3d1e6f480d53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:42.500620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.500511    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-credential-socket" pod "ef6dce00-fe43-484e-8d2f-3d1e6f480d53" (UID: "ef6dce00-fe43-484e-8d2f-3d1e6f480d53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:42.500620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.500531    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istiod-ca-cert" pod "ef6dce00-fe43-484e-8d2f-3d1e6f480d53" (UID: "ef6dce00-fe43-484e-8d2f-3d1e6f480d53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:42.500774 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.500646    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-workload-certs" pod "ef6dce00-fe43-484e-8d2f-3d1e6f480d53" (UID: "ef6dce00-fe43-484e-8d2f-3d1e6f480d53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:42.500859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.500841    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-token\" (UniqueName: \"kubernetes.io/projected/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-token\") pod \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") "
Oct 30 22:26:42.500914 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.500893    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-data\") pod \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\" (UID: \"ef6dce00-fe43-484e-8d2f-3d1e6f480d53\") "
Oct 30 22:26:42.501429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.501405    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:42.501610 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.501434    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:42.501610 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.501446    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:42.501610 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.501454    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:42.501610 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.501598    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-data" pod "ef6dce00-fe43-484e-8d2f-3d1e6f480d53" (UID: "ef6dce00-fe43-484e-8d2f-3d1e6f480d53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:42.502833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.502809    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-kube-api-access-bstql" pod "ef6dce00-fe43-484e-8d2f-3d1e6f480d53" (UID: "ef6dce00-fe43-484e-8d2f-3d1e6f480d53"). InnerVolumeSpecName "kube-api-access-bstql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:42.503216 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.503190    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-envoy" pod "ef6dce00-fe43-484e-8d2f-3d1e6f480d53" (UID: "ef6dce00-fe43-484e-8d2f-3d1e6f480d53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:42.503459 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.503438    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-podinfo" pod "ef6dce00-fe43-484e-8d2f-3d1e6f480d53" (UID: "ef6dce00-fe43-484e-8d2f-3d1e6f480d53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:26:42.503511 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.503457    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-token" pod "ef6dce00-fe43-484e-8d2f-3d1e6f480d53" (UID: "ef6dce00-fe43-484e-8d2f-3d1e6f480d53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:42.602831 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.602796    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:42.602831 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.602830    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:42.603070 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.602842    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:42.603070 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.602854    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:42.603070 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:42.602870    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bstql\" (UniqueName: \"kubernetes.io/projected/ef6dce00-fe43-484e-8d2f-3d1e6f480d53-kube-api-access-bstql\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:26:43.004833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:43.004730    4553 generic.go:356] "Generic (PLEG): container finished" podID="ef6dce00-fe43-484e-8d2f-3d1e6f480d53" containerID="c20c412bda9d26913bc233736452221022b6efbbdaefed610e7f4c88652c8a45" exitCode=0
Oct 30 22:26:43.004833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:43.004799    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"
Oct 30 22:26:43.004833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:43.004823    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6" event={"ID":"ef6dce00-fe43-484e-8d2f-3d1e6f480d53","Type":"ContainerDied","Data":"c20c412bda9d26913bc233736452221022b6efbbdaefed610e7f4c88652c8a45"}
Oct 30 22:26:43.005112 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:43.004903    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6" event={"ID":"ef6dce00-fe43-484e-8d2f-3d1e6f480d53","Type":"ContainerDied","Data":"e18800084acb85bd84884c26df7177d7e02987aa33e5628ae2210902f1f13973"}
Oct 30 22:26:43.005112 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:43.004925    4553 scope.go:122] "RemoveContainer" containerID="c20c412bda9d26913bc233736452221022b6efbbdaefed610e7f4c88652c8a45"
Oct 30 22:26:43.016534 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:43.016515    4553 scope.go:122] "RemoveContainer" containerID="c20c412bda9d26913bc233736452221022b6efbbdaefed610e7f4c88652c8a45"
Oct 30 22:26:43.016826 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:26:43.016806    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c20c412bda9d26913bc233736452221022b6efbbdaefed610e7f4c88652c8a45\": container with ID starting with c20c412bda9d26913bc233736452221022b6efbbdaefed610e7f4c88652c8a45 not found: ID does not exist" containerID="c20c412bda9d26913bc233736452221022b6efbbdaefed610e7f4c88652c8a45"
Oct 30 22:26:43.016889 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:43.016840    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c20c412bda9d26913bc233736452221022b6efbbdaefed610e7f4c88652c8a45"} err="failed to get container status \"c20c412bda9d26913bc233736452221022b6efbbdaefed610e7f4c88652c8a45\": rpc error: code = NotFound desc = could not find container \"c20c412bda9d26913bc233736452221022b6efbbdaefed610e7f4c88652c8a45\": container with ID starting with c20c412bda9d26913bc233736452221022b6efbbdaefed610e7f4c88652c8a45 not found: ID does not exist"
Oct 30 22:26:43.172881 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:43.172826    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"]
Oct 30 22:26:43.175856 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:43.175825    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-pqpw6"]
Oct 30 22:26:43.393695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:43.393656    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6dce00-fe43-484e-8d2f-3d1e6f480d53" path="/var/lib/kubelet/pods/ef6dce00-fe43-484e-8d2f-3d1e6f480d53/volumes"
Oct 30 22:26:45.992317 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:45.992286    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:26:51.039536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:51.039492    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:51.052543 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:51.052518    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:51.085925 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:26:51.085889    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:26:58.295839 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:26:58.295799    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:27:06.218302 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:06.218224    4553 generic.go:356] "Generic (PLEG): container finished" podID="c02574d3-4aa9-41f0-91c2-759b5d06654e" containerID="51ce6e237d8279cf53a6872a6d90773d0eb5b9055f1b1c9805452dfaf6614eec" exitCode=0
Oct 30 22:27:06.218812 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:06.218300    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8" event={"ID":"c02574d3-4aa9-41f0-91c2-759b5d06654e","Type":"ContainerDied","Data":"51ce6e237d8279cf53a6872a6d90773d0eb5b9055f1b1c9805452dfaf6614eec"}
Oct 30 22:27:06.218812 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:06.218779    4553 scope.go:122] "RemoveContainer" containerID="51ce6e237d8279cf53a6872a6d90773d0eb5b9055f1b1c9805452dfaf6614eec"
Oct 30 22:27:06.220242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:06.220223    4553 generic.go:356] "Generic (PLEG): container finished" podID="6a77b4ea-e62c-4f91-9264-d037dbe9210e" containerID="77d73cb21519996a3c577fe7a525a04b5d5cd26be8d2645ce40d781c3370170e" exitCode=0
Oct 30 22:27:06.220359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:06.220300    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc" event={"ID":"6a77b4ea-e62c-4f91-9264-d037dbe9210e","Type":"ContainerDied","Data":"77d73cb21519996a3c577fe7a525a04b5d5cd26be8d2645ce40d781c3370170e"}
Oct 30 22:27:06.220406 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:06.220346    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:27:06.220447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:06.220429    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:27:06.220494 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:06.220477    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:27:06.220724 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:06.220708    4553 scope.go:122] "RemoveContainer" containerID="77d73cb21519996a3c577fe7a525a04b5d5cd26be8d2645ce40d781c3370170e"
Oct 30 22:27:06.222044 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:06.222019    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:27:06.222104 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:06.222085    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:27:06.222167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:06.222119    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:27:07.263722 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:07.263682    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8" event={"ID":"c02574d3-4aa9-41f0-91c2-759b5d06654e","Type":"ContainerStarted","Data":"d7cd651a1f822ad298d4f4b74f241701265626c6a01757e4fdbbfeb7866e4c91"}
Oct 30 22:27:07.266246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:07.266220    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc" event={"ID":"6a77b4ea-e62c-4f91-9264-d037dbe9210e","Type":"ContainerStarted","Data":"0ec280cf0f4e5a8b58349f718d6b866e3db51dd564e623539534340a8136fc03"}
Oct 30 22:27:07.488170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:07.488111    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:27:07.490024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:07.489922    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:27:07.490024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:07.489998    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8" podUID="c02574d3-4aa9-41f0-91c2-759b5d06654e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 22:27:07.993295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:07.993251    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:27:07.993501 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:07.993382    4553 patch_prober.go:29] interesting pod/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:27:07.993501 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:07.993420    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc" podUID="6a77b4ea-e62c-4f91-9264-d037dbe9210e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 22:27:08.488082 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:08.488042    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:27:08.488605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:08.488097    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8" podUID="c02574d3-4aa9-41f0-91c2-759b5d06654e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 22:27:08.992317 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:08.992282    4553 patch_prober.go:29] interesting pod/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:27:08.992505 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:08.992336    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc" podUID="6a77b4ea-e62c-4f91-9264-d037dbe9210e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 22:27:09.487776 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:09.487729    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:27:09.487956 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:09.487822    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8" podUID="c02574d3-4aa9-41f0-91c2-759b5d06654e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 22:27:09.993175 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:09.993114    4553 patch_prober.go:29] interesting pod/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:27:09.993673 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:09.993634    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc" podUID="6a77b4ea-e62c-4f91-9264-d037dbe9210e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 22:27:10.482405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:10.482365    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:27:10.487676 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:10.487649    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:27:10.487829 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:10.487694    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8" podUID="c02574d3-4aa9-41f0-91c2-759b5d06654e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 22:27:10.992405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:10.992368    4553 patch_prober.go:29] interesting pod/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:27:10.992603 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:10.992432    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc" podUID="6a77b4ea-e62c-4f91-9264-d037dbe9210e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 22:27:11.487825 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:11.487784    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:27:11.488309 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:11.487846    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8" podUID="c02574d3-4aa9-41f0-91c2-759b5d06654e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 22:27:11.992819 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:11.992781    4553 patch_prober.go:29] interesting pod/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 22:27:11.993021 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:11.992856    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc" podUID="6a77b4ea-e62c-4f91-9264-d037dbe9210e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 22:27:12.487569 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:12.487522    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:27:12.487760 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:12.487595    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8" podUID="c02574d3-4aa9-41f0-91c2-759b5d06654e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 22:27:12.997704 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:12.997673    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:27:12.998193 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:12.998160    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:27:12.998762 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:12.998744    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:27:13.492384 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:13.492356    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:27:13.493340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:13.493316    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:27:25.668644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.668600    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"]
Oct 30 22:27:25.670284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.670200    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ef6dce00-fe43-484e-8d2f-3d1e6f480d53" containerName="istio-proxy"
Oct 30 22:27:25.670468 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.670452    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6dce00-fe43-484e-8d2f-3d1e6f480d53" containerName="istio-proxy"
Oct 30 22:27:25.670959 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.670936    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6dce00-fe43-484e-8d2f-3d1e6f480d53" containerName="istio-proxy"
Oct 30 22:27:25.675127 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.675103    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.676828 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.676800    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"]
Oct 30 22:27:25.681613 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.681584    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"
Oct 30 22:27:25.722260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.722227    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"]
Oct 30 22:27:25.772887 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.772844    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-42e8a1-dockercfg-qncd9\""
Oct 30 22:27:25.778024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.775205    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.778024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.775321    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.778024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.775353    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7h2q\" (UniqueName: \"kubernetes.io/projected/df13fb2a-c588-45a7-a5bc-b21a5b8aec6f-kube-api-access-q7h2q\") pod \"payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt\" (UID: \"df13fb2a-c588-45a7-a5bc-b21a5b8aec6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"
Oct 30 22:27:25.778024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.775397    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.778024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.775444    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.778024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.775481    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.778024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.775502    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtsk7\" (UniqueName: \"kubernetes.io/projected/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-kube-api-access-rtsk7\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.778024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.775547    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.778024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.775574    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.778024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.775611    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df13fb2a-c588-45a7-a5bc-b21a5b8aec6f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt\" (UID: \"df13fb2a-c588-45a7-a5bc-b21a5b8aec6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"
Oct 30 22:27:25.778024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.775650    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.781071 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.778973    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"]
Oct 30 22:27:25.793297 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.793268    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-42e8a1\""
Oct 30 22:27:25.819794 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.819758    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"]
Oct 30 22:27:25.824639 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.824617    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"
Oct 30 22:27:25.876512 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.876475    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.876699 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.876522    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.876699 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.876564    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df13fb2a-c588-45a7-a5bc-b21a5b8aec6f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt\" (UID: \"df13fb2a-c588-45a7-a5bc-b21a5b8aec6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"
Oct 30 22:27:25.876699 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.876607    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.876699 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.876693    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncprb\" (UniqueName: \"kubernetes.io/projected/6106ed06-a521-4806-8090-78724db50f2c-kube-api-access-ncprb\") pod \"payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz\" (UID: \"6106ed06-a521-4806-8090-78724db50f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"
Oct 30 22:27:25.876933 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.876778    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6106ed06-a521-4806-8090-78724db50f2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz\" (UID: \"6106ed06-a521-4806-8090-78724db50f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"
Oct 30 22:27:25.876933 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.876806    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.876933 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.876853    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.876933 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.876895    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7h2q\" (UniqueName: \"kubernetes.io/projected/df13fb2a-c588-45a7-a5bc-b21a5b8aec6f-kube-api-access-q7h2q\") pod \"payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt\" (UID: \"df13fb2a-c588-45a7-a5bc-b21a5b8aec6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"
Oct 30 22:27:25.877235 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.876961    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.877235 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.876966    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.877235 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.877036    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.877235 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.877086    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.877235 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.877111    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtsk7\" (UniqueName: \"kubernetes.io/projected/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-kube-api-access-rtsk7\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.878215 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.877432    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.878215 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.877475    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df13fb2a-c588-45a7-a5bc-b21a5b8aec6f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt\" (UID: \"df13fb2a-c588-45a7-a5bc-b21a5b8aec6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"
Oct 30 22:27:25.878215 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.877600    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.878215 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.877965    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.878215 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.878204    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.880529 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.880502    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.880811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.880789    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.882863 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.882837    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"]
Oct 30 22:27:25.935757 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.935686    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.936201 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.936176    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtsk7\" (UniqueName: \"kubernetes.io/projected/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-kube-api-access-rtsk7\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:25.937021 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.936996    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7h2q\" (UniqueName: \"kubernetes.io/projected/df13fb2a-c588-45a7-a5bc-b21a5b8aec6f-kube-api-access-q7h2q\") pod \"payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt\" (UID: \"df13fb2a-c588-45a7-a5bc-b21a5b8aec6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"
Oct 30 22:27:25.978316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.978279    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncprb\" (UniqueName: \"kubernetes.io/projected/6106ed06-a521-4806-8090-78724db50f2c-kube-api-access-ncprb\") pod \"payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz\" (UID: \"6106ed06-a521-4806-8090-78724db50f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"
Oct 30 22:27:25.978495 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.978363    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6106ed06-a521-4806-8090-78724db50f2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz\" (UID: \"6106ed06-a521-4806-8090-78724db50f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"
Oct 30 22:27:25.979115 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:25.979090    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6106ed06-a521-4806-8090-78724db50f2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz\" (UID: \"6106ed06-a521-4806-8090-78724db50f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"
Oct 30 22:27:26.006585 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:26.006545    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:26.012693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:26.012668    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"
Oct 30 22:27:26.037214 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:26.037185    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncprb\" (UniqueName: \"kubernetes.io/projected/6106ed06-a521-4806-8090-78724db50f2c-kube-api-access-ncprb\") pod \"payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz\" (UID: \"6106ed06-a521-4806-8090-78724db50f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"
Oct 30 22:27:26.141288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:26.141254    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"
Oct 30 22:27:26.444822 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:26.444793    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"]
Oct 30 22:27:26.446651 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:27:26.446624    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf13fb2a_c588_45a7_a5bc_b21a5b8aec6f.slice/crio-7f460e4429e953269bc6c56e3efd62d9690d9fe2d8a02ab02a155d88cca5be8b WatchSource:0}: Error finding container 7f460e4429e953269bc6c56e3efd62d9690d9fe2d8a02ab02a155d88cca5be8b: Status 404 returned error can't find the container with id 7f460e4429e953269bc6c56e3efd62d9690d9fe2d8a02ab02a155d88cca5be8b
Oct 30 22:27:26.447480 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:27:26.447445    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb81c50_78ea_4df4_9b3c_3525e293d8c0.slice/crio-8594b19746d67ff9afa2720df3f8adb1677616874b3956b32d1aaab66fbe9924 WatchSource:0}: Error finding container 8594b19746d67ff9afa2720df3f8adb1677616874b3956b32d1aaab66fbe9924: Status 404 returned error can't find the container with id 8594b19746d67ff9afa2720df3f8adb1677616874b3956b32d1aaab66fbe9924
Oct 30 22:27:26.448277 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:26.448253    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"]
Oct 30 22:27:26.450363 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:26.450300    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:27:26.450455 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:26.450384    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:27:26.450455 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:26.450426    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:27:26.549112 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:26.549077    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"]
Oct 30 22:27:26.550611 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:27:26.550580    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6106ed06_a521_4806_8090_78724db50f2c.slice/crio-d0ba39666bf404f12ed6dc0a48f1e8383c73fcab7ed620498955c30af55873f6 WatchSource:0}: Error finding container d0ba39666bf404f12ed6dc0a48f1e8383c73fcab7ed620498955c30af55873f6: Status 404 returned error can't find the container with id d0ba39666bf404f12ed6dc0a48f1e8383c73fcab7ed620498955c30af55873f6
Oct 30 22:27:27.396691 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:27.396655    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls" event={"ID":"5bb81c50-78ea-4df4-9b3c-3525e293d8c0","Type":"ContainerStarted","Data":"ddda3111ff48aa64f7c974f02683c9b12b92fa6619509eda26de6e78a419e37b"}
Oct 30 22:27:27.397241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:27.396714    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls" event={"ID":"5bb81c50-78ea-4df4-9b3c-3525e293d8c0","Type":"ContainerStarted","Data":"8594b19746d67ff9afa2720df3f8adb1677616874b3956b32d1aaab66fbe9924"}
Oct 30 22:27:27.398420 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:27.398392    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz" event={"ID":"6106ed06-a521-4806-8090-78724db50f2c","Type":"ContainerStarted","Data":"cb8654ed9090d37b201e5124002382b6b76a280b73304a2557e4f0a2ab9f8640"}
Oct 30 22:27:27.398571 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:27.398443    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz" event={"ID":"6106ed06-a521-4806-8090-78724db50f2c","Type":"ContainerStarted","Data":"d0ba39666bf404f12ed6dc0a48f1e8383c73fcab7ed620498955c30af55873f6"}
Oct 30 22:27:27.398571 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:27.398475    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"
Oct 30 22:27:27.399999 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:27.399979    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt" event={"ID":"df13fb2a-c588-45a7-a5bc-b21a5b8aec6f","Type":"ContainerStarted","Data":"eaebea7c7144475446153503a9300931ce9189b657d5ab9ac72c648045eef33e"}
Oct 30 22:27:27.400079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:27.400013    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt" event={"ID":"df13fb2a-c588-45a7-a5bc-b21a5b8aec6f","Type":"ContainerStarted","Data":"7f460e4429e953269bc6c56e3efd62d9690d9fe2d8a02ab02a155d88cca5be8b"}
Oct 30 22:27:27.400079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:27.400056    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"
Oct 30 22:27:27.516694 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:27.516629    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls" podStartSLOduration=2.516612891 podStartE2EDuration="2.516612891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:27.515778686 +0000 UTC m=+2698.250141717" watchObservedRunningTime="2026-10-30 22:27:27.516612891 +0000 UTC m=+2698.250975923"
Oct 30 22:27:27.636861 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:27.636799    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz" podStartSLOduration=2.636782936 podStartE2EDuration="2.636782936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:27.634854383 +0000 UTC m=+2698.369217416" watchObservedRunningTime="2026-10-30 22:27:27.636782936 +0000 UTC m=+2698.371145967"
Oct 30 22:27:27.751710 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:27.751588    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt" podStartSLOduration=2.751572029 podStartE2EDuration="2.751572029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:27.748550899 +0000 UTC m=+2698.482913931" watchObservedRunningTime="2026-10-30 22:27:27.751572029 +0000 UTC m=+2698.485935060"
Oct 30 22:27:28.006884 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:28.006801    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:28.030224 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:28.008262    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:27:28.030224 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:28.008320    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls" podUID="5bb81c50-78ea-4df4-9b3c-3525e293d8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 22:27:28.251097 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:27:28.251058    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:27:29.007697 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:29.007662    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:27:29.008210 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:29.007735    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls" podUID="5bb81c50-78ea-4df4-9b3c-3525e293d8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 22:27:29.871084 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:29.871053    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:27:29.874241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:29.874217    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:27:30.007389 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:30.007351    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:27:30.007577 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:30.007409    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls" podUID="5bb81c50-78ea-4df4-9b3c-3525e293d8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 22:27:31.007099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:31.007063    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:27:31.007559 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:31.007115    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls" podUID="5bb81c50-78ea-4df4-9b3c-3525e293d8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 22:27:32.007782 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:32.007746    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:27:32.008282 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:32.007804    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls" podUID="5bb81c50-78ea-4df4-9b3c-3525e293d8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 22:27:33.007322 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:33.007292    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body=
Oct 30 22:27:33.007509 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:33.007350    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls" podUID="5bb81c50-78ea-4df4-9b3c-3525e293d8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused"
Oct 30 22:27:35.048353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:35.048309    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:27:35.048945 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:35.048400    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls" podUID="5bb81c50-78ea-4df4-9b3c-3525e293d8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:27:35.160514 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:35.160485    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:35.160992 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:35.160963    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:35.162627 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:35.162604    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:27:38.410658 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:38.410634    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"
Oct 30 22:27:38.411077 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:38.410707    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"
Oct 30 22:27:52.771621 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:52.771583    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"]
Oct 30 22:27:52.778156 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:52.778113    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"
Oct 30 22:27:52.821959 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:52.821923    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"]
Oct 30 22:27:52.822301 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:52.822185    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7" podUID="a3a2bd94-6485-452d-98da-b656035143cb" containerName="istio-proxy" containerID="cri-o://9bfeaaec1b1967dc7bf0cba8305d2e3deee24e3936d140cd99496f58d60f07a2" gracePeriod=30
Oct 30 22:27:52.835079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:52.835044    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"]
Oct 30 22:27:52.837192 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:52.837120    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b563fd\""
Oct 30 22:27:52.837719 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:52.837695    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b563fd-dockercfg-fpnl8\""
Oct 30 22:27:52.841362 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:52.841332    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4zgk\" (UniqueName: \"kubernetes.io/projected/23f5c585-dd28-4678-8318-22ebc5a69e77-kube-api-access-m4zgk\") pod \"payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm\" (UID: \"23f5c585-dd28-4678-8318-22ebc5a69e77\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"
Oct 30 22:27:52.841505 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:52.841457    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f5c585-dd28-4678-8318-22ebc5a69e77-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm\" (UID: \"23f5c585-dd28-4678-8318-22ebc5a69e77\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"
Oct 30 22:27:52.883308 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:52.883268    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"]
Oct 30 22:27:52.887788 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:52.887762    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"
Oct 30 22:27:52.958999 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:52.958952    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f5c585-dd28-4678-8318-22ebc5a69e77-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm\" (UID: \"23f5c585-dd28-4678-8318-22ebc5a69e77\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"
Oct 30 22:27:52.959298 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:52.959210    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4zgk\" (UniqueName: \"kubernetes.io/projected/23f5c585-dd28-4678-8318-22ebc5a69e77-kube-api-access-m4zgk\") pod \"payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm\" (UID: \"23f5c585-dd28-4678-8318-22ebc5a69e77\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"
Oct 30 22:27:52.964185 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:52.960517    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f5c585-dd28-4678-8318-22ebc5a69e77-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm\" (UID: \"23f5c585-dd28-4678-8318-22ebc5a69e77\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"
Oct 30 22:27:52.981465 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:52.981428    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"]
Oct 30 22:27:53.019098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:53.019068    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4zgk\" (UniqueName: \"kubernetes.io/projected/23f5c585-dd28-4678-8318-22ebc5a69e77-kube-api-access-m4zgk\") pod \"payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm\" (UID: \"23f5c585-dd28-4678-8318-22ebc5a69e77\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"
Oct 30 22:27:53.060978 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:53.060890    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64mq4\" (UniqueName: \"kubernetes.io/projected/025c2f88-9c30-4930-93ca-4a3305dbc53e-kube-api-access-64mq4\") pod \"payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf\" (UID: \"025c2f88-9c30-4930-93ca-4a3305dbc53e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"
Oct 30 22:27:53.060978 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:53.060934    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025c2f88-9c30-4930-93ca-4a3305dbc53e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf\" (UID: \"025c2f88-9c30-4930-93ca-4a3305dbc53e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"
Oct 30 22:27:53.093436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:53.093337    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"
Oct 30 22:27:53.161993 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:53.161956    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mq4\" (UniqueName: \"kubernetes.io/projected/025c2f88-9c30-4930-93ca-4a3305dbc53e-kube-api-access-64mq4\") pod \"payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf\" (UID: \"025c2f88-9c30-4930-93ca-4a3305dbc53e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"
Oct 30 22:27:53.162098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:53.162014    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025c2f88-9c30-4930-93ca-4a3305dbc53e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf\" (UID: \"025c2f88-9c30-4930-93ca-4a3305dbc53e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"
Oct 30 22:27:53.164532 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:53.164501    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025c2f88-9c30-4930-93ca-4a3305dbc53e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf\" (UID: \"025c2f88-9c30-4930-93ca-4a3305dbc53e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"
Oct 30 22:27:53.221433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:53.221390    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mq4\" (UniqueName: \"kubernetes.io/projected/025c2f88-9c30-4930-93ca-4a3305dbc53e-kube-api-access-64mq4\") pod \"payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf\" (UID: \"025c2f88-9c30-4930-93ca-4a3305dbc53e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"
Oct 30 22:27:53.474010 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:53.473977    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"]
Oct 30 22:27:53.475336 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:27:53.475311    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f5c585_dd28_4678_8318_22ebc5a69e77.slice/crio-95b9c03c218e21cbf9eb1cd22bbf44439a91cd09741eba206dc6876fe3985e40 WatchSource:0}: Error finding container 95b9c03c218e21cbf9eb1cd22bbf44439a91cd09741eba206dc6876fe3985e40: Status 404 returned error can't find the container with id 95b9c03c218e21cbf9eb1cd22bbf44439a91cd09741eba206dc6876fe3985e40
Oct 30 22:27:53.502631 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:53.502606    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"
Oct 30 22:27:53.701351 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:53.701309    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm" event={"ID":"23f5c585-dd28-4678-8318-22ebc5a69e77","Type":"ContainerStarted","Data":"996501539dfc144b7da2566b0a0003e1d10f737e5402836e592b841b86e5259b"}
Oct 30 22:27:53.701543 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:53.701383    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm" event={"ID":"23f5c585-dd28-4678-8318-22ebc5a69e77","Type":"ContainerStarted","Data":"95b9c03c218e21cbf9eb1cd22bbf44439a91cd09741eba206dc6876fe3985e40"}
Oct 30 22:27:53.701543 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:53.701411    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"
Oct 30 22:27:53.824207 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:53.824078    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm" podStartSLOduration=1.824059461 podStartE2EDuration="1.824059461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:53.819911878 +0000 UTC m=+2724.554274909" watchObservedRunningTime="2026-10-30 22:27:53.824059461 +0000 UTC m=+2724.558422494"
Oct 30 22:27:53.889859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:53.889830    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"]
Oct 30 22:27:53.890637 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:27:53.890608    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025c2f88_9c30_4930_93ca_4a3305dbc53e.slice/crio-678e12af6ece389c09bce1d30ed47ef54fcccfa8f333153e4840a94c7738120b WatchSource:0}: Error finding container 678e12af6ece389c09bce1d30ed47ef54fcccfa8f333153e4840a94c7738120b: Status 404 returned error can't find the container with id 678e12af6ece389c09bce1d30ed47ef54fcccfa8f333153e4840a94c7738120b
Oct 30 22:27:54.709558 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:54.709514    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf" event={"ID":"025c2f88-9c30-4930-93ca-4a3305dbc53e","Type":"ContainerStarted","Data":"85361e16c4f8f998f91b1a23ba60468e76171cbdf24708e613b918e2fd578154"}
Oct 30 22:27:54.709807 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:54.709784    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"
Oct 30 22:27:54.709877 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:54.709834    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf" event={"ID":"025c2f88-9c30-4930-93ca-4a3305dbc53e","Type":"ContainerStarted","Data":"678e12af6ece389c09bce1d30ed47ef54fcccfa8f333153e4840a94c7738120b"}
Oct 30 22:27:54.827325 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:54.827280    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf" podStartSLOduration=2.827266035 podStartE2EDuration="2.827266035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:54.825352669 +0000 UTC m=+2725.559715700" watchObservedRunningTime="2026-10-30 22:27:54.827266035 +0000 UTC m=+2725.561629067"
Oct 30 22:27:55.640209 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.640173    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"]
Oct 30 22:27:55.644719 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.644701    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.694616 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.694587    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.694790 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.694737    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.694837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.694793    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616b564b-7078-4e2d-9111-980e22584acd-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.694897 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.694853    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616b564b-7078-4e2d-9111-980e22584acd-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.694897 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.694892    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlt6z\" (UniqueName: \"kubernetes.io/projected/616b564b-7078-4e2d-9111-980e22584acd-kube-api-access-tlt6z\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.694979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.694935    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.695027 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.695013    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.695065 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.695042    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616b564b-7078-4e2d-9111-980e22584acd-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.695110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.695071    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.751835 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.751787    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"]
Oct 30 22:27:55.796599 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.796558    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.796820 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.796607    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616b564b-7078-4e2d-9111-980e22584acd-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.796820 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.796681    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616b564b-7078-4e2d-9111-980e22584acd-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.796820 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.796735    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlt6z\" (UniqueName: \"kubernetes.io/projected/616b564b-7078-4e2d-9111-980e22584acd-kube-api-access-tlt6z\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.796820 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.796773    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.797092 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.796825    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.797092 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.796857    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616b564b-7078-4e2d-9111-980e22584acd-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.797092 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.796892    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.797092 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.796948    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.797350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.797126    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-workload-certs\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.797350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.797150    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-credential-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.797926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.797854    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-istio-data\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.798044 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.797960    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-workload-socket\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.798203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.798179    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616b564b-7078-4e2d-9111-980e22584acd-istiod-ca-cert\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.799955 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.799934    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616b564b-7078-4e2d-9111-980e22584acd-istio-podinfo\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.800280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.800262    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-istio-envoy\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.856481 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.856452    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616b564b-7078-4e2d-9111-980e22584acd-istio-token\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.858054 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.858034    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlt6z\" (UniqueName: \"kubernetes.io/projected/616b564b-7078-4e2d-9111-980e22584acd-kube-api-access-tlt6z\") pod \"e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") " pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:55.960220 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:55.960098    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:56.405901 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:56.405777    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"]
Oct 30 22:27:56.412744 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:27:56.412700    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616b564b_7078_4e2d_9111_980e22584acd.slice/crio-d48af2281888c617905362a225767860ec96828b60d882a443ba778c7ae10b2c WatchSource:0}: Error finding container d48af2281888c617905362a225767860ec96828b60d882a443ba778c7ae10b2c: Status 404 returned error can't find the container with id d48af2281888c617905362a225767860ec96828b60d882a443ba778c7ae10b2c
Oct 30 22:27:56.415764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:56.415721    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:27:56.415899 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:56.415813    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:27:56.415899 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:56.415854    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:27:56.681557 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:56.681457    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-c8sl7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:27:56.681557 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:56.681519    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7" podUID="a3a2bd94-6485-452d-98da-b656035143cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:27:56.726721 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:56.726672    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h" event={"ID":"616b564b-7078-4e2d-9111-980e22584acd","Type":"ContainerStarted","Data":"27b073bd9b353a16a666e886392017c55b3a9709ed33ca84d216573bc662bc75"}
Oct 30 22:27:56.726937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:56.726752    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h" event={"ID":"616b564b-7078-4e2d-9111-980e22584acd","Type":"ContainerStarted","Data":"d48af2281888c617905362a225767860ec96828b60d882a443ba778c7ae10b2c"}
Oct 30 22:27:56.855296 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:56.855246    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h" podStartSLOduration=1.8552253429999999 podStartE2EDuration="1.855225343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:56.85484992 +0000 UTC m=+2727.589212952" watchObservedRunningTime="2026-10-30 22:27:56.855225343 +0000 UTC m=+2727.589588423"
Oct 30 22:27:56.964087 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:56.963956    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:27:57.972626 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:57.972598    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 22:27:57.973204 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:57.973171    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h" podUID="616b564b-7078-4e2d-9111-980e22584acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 22:27:58.204217 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:27:58.204187    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:27:58.219659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.219335    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:27:58.269166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.269112    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-istio-envoy\") pod \"a3a2bd94-6485-452d-98da-b656035143cb\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") "
Oct 30 22:27:58.269342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.269203    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a2bd94-6485-452d-98da-b656035143cb-kube-api-access-pvc5w\" (UniqueName: \"kubernetes.io/projected/a3a2bd94-6485-452d-98da-b656035143cb-kube-api-access-pvc5w\") pod \"a3a2bd94-6485-452d-98da-b656035143cb\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") "
Oct 30 22:27:58.269342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.269240    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a2bd94-6485-452d-98da-b656035143cb-istio-token\" (UniqueName: \"kubernetes.io/projected/a3a2bd94-6485-452d-98da-b656035143cb-istio-token\") pod \"a3a2bd94-6485-452d-98da-b656035143cb\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") "
Oct 30 22:27:58.269342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.269264    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a2bd94-6485-452d-98da-b656035143cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a2bd94-6485-452d-98da-b656035143cb-istiod-ca-cert\") pod \"a3a2bd94-6485-452d-98da-b656035143cb\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") "
Oct 30 22:27:58.269342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.269332    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-istio-data\") pod \"a3a2bd94-6485-452d-98da-b656035143cb\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") "
Oct 30 22:27:58.269556 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.269419    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-workload-socket\") pod \"a3a2bd94-6485-452d-98da-b656035143cb\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") "
Oct 30 22:27:58.269556 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.269454    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3a2bd94-6485-452d-98da-b656035143cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a2bd94-6485-452d-98da-b656035143cb-istio-podinfo\") pod \"a3a2bd94-6485-452d-98da-b656035143cb\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") "
Oct 30 22:27:58.269556 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.269524    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-credential-socket\") pod \"a3a2bd94-6485-452d-98da-b656035143cb\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") "
Oct 30 22:27:58.269691 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.269569    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-workload-certs\") pod \"a3a2bd94-6485-452d-98da-b656035143cb\" (UID: \"a3a2bd94-6485-452d-98da-b656035143cb\") "
Oct 30 22:27:58.269975 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.269820    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-workload-socket" pod "a3a2bd94-6485-452d-98da-b656035143cb" (UID: "a3a2bd94-6485-452d-98da-b656035143cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:58.269975 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.269833    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a2bd94-6485-452d-98da-b656035143cb-istiod-ca-cert" pod "a3a2bd94-6485-452d-98da-b656035143cb" (UID: "a3a2bd94-6485-452d-98da-b656035143cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:58.269975 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.269939    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-istio-data" pod "a3a2bd94-6485-452d-98da-b656035143cb" (UID: "a3a2bd94-6485-452d-98da-b656035143cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:58.269975 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.269937    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-credential-socket" pod "a3a2bd94-6485-452d-98da-b656035143cb" (UID: "a3a2bd94-6485-452d-98da-b656035143cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:58.270333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.270060    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-workload-certs" pod "a3a2bd94-6485-452d-98da-b656035143cb" (UID: "a3a2bd94-6485-452d-98da-b656035143cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:58.270394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.270349    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:58.270394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.270372    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:58.270394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.270387    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:58.270525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.270405    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3a2bd94-6485-452d-98da-b656035143cb-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:58.270525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.270422    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:58.271643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.271619    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a2bd94-6485-452d-98da-b656035143cb-kube-api-access-pvc5w" pod "a3a2bd94-6485-452d-98da-b656035143cb" (UID: "a3a2bd94-6485-452d-98da-b656035143cb"). InnerVolumeSpecName "kube-api-access-pvc5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:58.272185 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.272157    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3a2bd94-6485-452d-98da-b656035143cb-istio-podinfo" pod "a3a2bd94-6485-452d-98da-b656035143cb" (UID: "a3a2bd94-6485-452d-98da-b656035143cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:58.272185 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.272166    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-istio-envoy" pod "a3a2bd94-6485-452d-98da-b656035143cb" (UID: "a3a2bd94-6485-452d-98da-b656035143cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:58.272316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.272197    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a2bd94-6485-452d-98da-b656035143cb-istio-token" pod "a3a2bd94-6485-452d-98da-b656035143cb" (UID: "a3a2bd94-6485-452d-98da-b656035143cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:58.371830 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.371797    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3a2bd94-6485-452d-98da-b656035143cb-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:58.371830 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.371828    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3a2bd94-6485-452d-98da-b656035143cb-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:58.372033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.371839    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3a2bd94-6485-452d-98da-b656035143cb-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:58.372033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.371848    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvc5w\" (UniqueName: \"kubernetes.io/projected/a3a2bd94-6485-452d-98da-b656035143cb-kube-api-access-pvc5w\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:27:58.839057 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.839022    4553 generic.go:356] "Generic (PLEG): container finished" podID="a3a2bd94-6485-452d-98da-b656035143cb" containerID="9bfeaaec1b1967dc7bf0cba8305d2e3deee24e3936d140cd99496f58d60f07a2" exitCode=0
Oct 30 22:27:58.839312 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.839180    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"
Oct 30 22:27:58.839312 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.839128    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7" event={"ID":"a3a2bd94-6485-452d-98da-b656035143cb","Type":"ContainerDied","Data":"9bfeaaec1b1967dc7bf0cba8305d2e3deee24e3936d140cd99496f58d60f07a2"}
Oct 30 22:27:58.839312 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.839271    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7" event={"ID":"a3a2bd94-6485-452d-98da-b656035143cb","Type":"ContainerDied","Data":"9bc5d016ed0c55ce15a86567f35e741d416e198a694839800cc51e50540871b8"}
Oct 30 22:27:58.839312 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.839299    4553 scope.go:122] "RemoveContainer" containerID="9bfeaaec1b1967dc7bf0cba8305d2e3deee24e3936d140cd99496f58d60f07a2"
Oct 30 22:27:58.854254 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.854231    4553 scope.go:122] "RemoveContainer" containerID="9bfeaaec1b1967dc7bf0cba8305d2e3deee24e3936d140cd99496f58d60f07a2"
Oct 30 22:27:58.854564 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:27:58.854543    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bfeaaec1b1967dc7bf0cba8305d2e3deee24e3936d140cd99496f58d60f07a2\": container with ID starting with 9bfeaaec1b1967dc7bf0cba8305d2e3deee24e3936d140cd99496f58d60f07a2 not found: ID does not exist" containerID="9bfeaaec1b1967dc7bf0cba8305d2e3deee24e3936d140cd99496f58d60f07a2"
Oct 30 22:27:58.854625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.854572    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bfeaaec1b1967dc7bf0cba8305d2e3deee24e3936d140cd99496f58d60f07a2"} err="failed to get container status \"9bfeaaec1b1967dc7bf0cba8305d2e3deee24e3936d140cd99496f58d60f07a2\": rpc error: code = NotFound desc = could not find container \"9bfeaaec1b1967dc7bf0cba8305d2e3deee24e3936d140cd99496f58d60f07a2\": container with ID starting with 9bfeaaec1b1967dc7bf0cba8305d2e3deee24e3936d140cd99496f58d60f07a2 not found: ID does not exist"
Oct 30 22:27:58.960856 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.960816    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 22:27:58.961066 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:58.960878    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h" podUID="616b564b-7078-4e2d-9111-980e22584acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 22:27:59.031160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:59.027187    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"]
Oct 30 22:27:59.058951 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:59.058920    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-c8sl7"]
Oct 30 22:27:59.394418 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:59.394386    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a2bd94-6485-452d-98da-b656035143cb" path="/var/lib/kubelet/pods/a3a2bd94-6485-452d-98da-b656035143cb/volumes"
Oct 30 22:27:59.960485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:59.960449    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 22:27:59.960650 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:27:59.960538    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h" podUID="616b564b-7078-4e2d-9111-980e22584acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 22:28:00.960793 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:00.960762    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 22:28:00.961217 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:00.960822    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h" podUID="616b564b-7078-4e2d-9111-980e22584acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 22:28:01.960448 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:01.960407    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 22:28:01.960677 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:01.960489    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h" podUID="616b564b-7078-4e2d-9111-980e22584acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 22:28:02.961585 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:02.961540    4553 patch_prober.go:29] interesting pod/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Oct 30 22:28:02.962049 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:02.961619    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h" podUID="616b564b-7078-4e2d-9111-980e22584acd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Oct 30 22:28:04.107944 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:04.107911    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:28:04.108901 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:04.108876    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:28:04.109562 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:04.109540    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:28:04.713168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:04.713125    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"
Oct 30 22:28:05.719771 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:05.719737    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"
Oct 30 22:28:07.765674 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:07.765631    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"]
Oct 30 22:28:12.766535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:12.766425    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk" podUID="48f440cc-1942-4ccd-92d6-936b1c59ac30" containerName="payload-pre-processing" containerID="cri-o://d1796f69cdaff3bfc4a33e2e154994fa5c65128a2015de02b32584c1fe1911d7" gracePeriod=25
Oct 30 22:28:12.939556 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:12.939527    4553 generic.go:356] "Generic (PLEG): container finished" podID="48f440cc-1942-4ccd-92d6-936b1c59ac30" containerID="d1796f69cdaff3bfc4a33e2e154994fa5c65128a2015de02b32584c1fe1911d7" exitCode=0
Oct 30 22:28:12.939720 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:12.939642    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk" event={"ID":"48f440cc-1942-4ccd-92d6-936b1c59ac30","Type":"ContainerDied","Data":"d1796f69cdaff3bfc4a33e2e154994fa5c65128a2015de02b32584c1fe1911d7"}
Oct 30 22:28:13.051338 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:13.051308    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"
Oct 30 22:28:13.105064 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:13.105031    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f440cc-1942-4ccd-92d6-936b1c59ac30-kube-api-access-7w726\" (UniqueName: \"kubernetes.io/projected/48f440cc-1942-4ccd-92d6-936b1c59ac30-kube-api-access-7w726\") pod \"48f440cc-1942-4ccd-92d6-936b1c59ac30\" (UID: \"48f440cc-1942-4ccd-92d6-936b1c59ac30\") "
Oct 30 22:28:13.105269 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:13.105187    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48f440cc-1942-4ccd-92d6-936b1c59ac30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f440cc-1942-4ccd-92d6-936b1c59ac30-plugins-config-volume\") pod \"48f440cc-1942-4ccd-92d6-936b1c59ac30\" (UID: \"48f440cc-1942-4ccd-92d6-936b1c59ac30\") "
Oct 30 22:28:13.105533 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:13.105514    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f440cc-1942-4ccd-92d6-936b1c59ac30-plugins-config-volume" pod "48f440cc-1942-4ccd-92d6-936b1c59ac30" (UID: "48f440cc-1942-4ccd-92d6-936b1c59ac30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:13.105695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:13.105681    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f440cc-1942-4ccd-92d6-936b1c59ac30-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:13.107556 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:13.107536    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f440cc-1942-4ccd-92d6-936b1c59ac30-kube-api-access-7w726" pod "48f440cc-1942-4ccd-92d6-936b1c59ac30" (UID: "48f440cc-1942-4ccd-92d6-936b1c59ac30"). InnerVolumeSpecName "kube-api-access-7w726". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:13.206882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:13.206852    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w726\" (UniqueName: \"kubernetes.io/projected/48f440cc-1942-4ccd-92d6-936b1c59ac30-kube-api-access-7w726\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:13.947525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:13.947492    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk" event={"ID":"48f440cc-1942-4ccd-92d6-936b1c59ac30","Type":"ContainerDied","Data":"2c70e9daadfb5321f7f86f61ecdc4483bbada8a22248fca643e8bc8b767e25de"}
Oct 30 22:28:13.947991 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:13.947548    4553 scope.go:122] "RemoveContainer" containerID="d1796f69cdaff3bfc4a33e2e154994fa5c65128a2015de02b32584c1fe1911d7"
Oct 30 22:28:13.947991 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:13.947500    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"
Oct 30 22:28:14.119909 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.119872    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"]
Oct 30 22:28:14.168787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.168748    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cce50a-d7946cc79-g5nnk"]
Oct 30 22:28:14.218728 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.218634    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"]
Oct 30 22:28:14.218891 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.218845    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7" podUID="6bbfe405-12a5-4eae-851a-8b2fc99947a0" containerName="maas-api" containerID="cri-o://42298b409eee7424f1c4fc451e5b701a97efdbad7d6c33069d707d2900d04f80" gracePeriod=30
Oct 30 22:28:14.507519 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.507495    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:28:14.521793 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.521766    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bbfe405-12a5-4eae-851a-8b2fc99947a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bbfe405-12a5-4eae-851a-8b2fc99947a0-openshift-service-ca\") pod \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") "
Oct 30 22:28:14.521936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.521867    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbfe405-12a5-4eae-851a-8b2fc99947a0-kube-api-access-6hcgk\" (UniqueName: \"kubernetes.io/projected/6bbfe405-12a5-4eae-851a-8b2fc99947a0-kube-api-access-6hcgk\") pod \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") "
Oct 30 22:28:14.521936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.521915    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-maas-api-tls\") pod \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") "
Oct 30 22:28:14.522062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.521969    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-metrics-certs\") pod \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\" (UID: \"6bbfe405-12a5-4eae-851a-8b2fc99947a0\") "
Oct 30 22:28:14.522252 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.522219    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbfe405-12a5-4eae-851a-8b2fc99947a0-openshift-service-ca" pod "6bbfe405-12a5-4eae-851a-8b2fc99947a0" (UID: "6bbfe405-12a5-4eae-851a-8b2fc99947a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:14.522399 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.522325    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bbfe405-12a5-4eae-851a-8b2fc99947a0-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:14.524367 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.524336    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-metrics-certs" pod "6bbfe405-12a5-4eae-851a-8b2fc99947a0" (UID: "6bbfe405-12a5-4eae-851a-8b2fc99947a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:14.524467 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.524421    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-maas-api-tls" pod "6bbfe405-12a5-4eae-851a-8b2fc99947a0" (UID: "6bbfe405-12a5-4eae-851a-8b2fc99947a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:14.524467 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.524455    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbfe405-12a5-4eae-851a-8b2fc99947a0-kube-api-access-6hcgk" pod "6bbfe405-12a5-4eae-851a-8b2fc99947a0" (UID: "6bbfe405-12a5-4eae-851a-8b2fc99947a0"). InnerVolumeSpecName "kube-api-access-6hcgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:14.627260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.627214    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hcgk\" (UniqueName: \"kubernetes.io/projected/6bbfe405-12a5-4eae-851a-8b2fc99947a0-kube-api-access-6hcgk\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:14.627260 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.627260    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:14.627605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.627278    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bbfe405-12a5-4eae-851a-8b2fc99947a0-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:14.955804 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.955774    4553 generic.go:356] "Generic (PLEG): container finished" podID="6bbfe405-12a5-4eae-851a-8b2fc99947a0" containerID="42298b409eee7424f1c4fc451e5b701a97efdbad7d6c33069d707d2900d04f80" exitCode=0
Oct 30 22:28:14.956316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.955821    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"
Oct 30 22:28:14.956316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.955868    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7" event={"ID":"6bbfe405-12a5-4eae-851a-8b2fc99947a0","Type":"ContainerDied","Data":"42298b409eee7424f1c4fc451e5b701a97efdbad7d6c33069d707d2900d04f80"}
Oct 30 22:28:14.956316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.955922    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7" event={"ID":"6bbfe405-12a5-4eae-851a-8b2fc99947a0","Type":"ContainerDied","Data":"c25a87ded4327a5aafede1352277116922be589b6c44951b7a9d0fcf84ea3262"}
Oct 30 22:28:14.956316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.955950    4553 scope.go:122] "RemoveContainer" containerID="42298b409eee7424f1c4fc451e5b701a97efdbad7d6c33069d707d2900d04f80"
Oct 30 22:28:14.967426 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.967407    4553 scope.go:122] "RemoveContainer" containerID="42298b409eee7424f1c4fc451e5b701a97efdbad7d6c33069d707d2900d04f80"
Oct 30 22:28:14.967728 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:28:14.967707    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42298b409eee7424f1c4fc451e5b701a97efdbad7d6c33069d707d2900d04f80\": container with ID starting with 42298b409eee7424f1c4fc451e5b701a97efdbad7d6c33069d707d2900d04f80 not found: ID does not exist" containerID="42298b409eee7424f1c4fc451e5b701a97efdbad7d6c33069d707d2900d04f80"
Oct 30 22:28:14.967814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:14.967736    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42298b409eee7424f1c4fc451e5b701a97efdbad7d6c33069d707d2900d04f80"} err="failed to get container status \"42298b409eee7424f1c4fc451e5b701a97efdbad7d6c33069d707d2900d04f80\": rpc error: code = NotFound desc = could not find container \"42298b409eee7424f1c4fc451e5b701a97efdbad7d6c33069d707d2900d04f80\": container with ID starting with 42298b409eee7424f1c4fc451e5b701a97efdbad7d6c33069d707d2900d04f80 not found: ID does not exist"
Oct 30 22:28:15.132120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.130627    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"]
Oct 30 22:28:15.140188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.140118    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-cce50a-6d7574fb58-x6fq7"]
Oct 30 22:28:15.394195 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.394156    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48f440cc-1942-4ccd-92d6-936b1c59ac30" path="/var/lib/kubelet/pods/48f440cc-1942-4ccd-92d6-936b1c59ac30/volumes"
Oct 30 22:28:15.394651 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.394637    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bbfe405-12a5-4eae-851a-8b2fc99947a0" path="/var/lib/kubelet/pods/6bbfe405-12a5-4eae-851a-8b2fc99947a0/volumes"
Oct 30 22:28:15.871983 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.871951    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"]
Oct 30 22:28:15.872609 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.872596    4553 cpu_manager.go:414] "containerMap: removing container" podUID="48f440cc-1942-4ccd-92d6-936b1c59ac30" containerName="payload-pre-processing"
Oct 30 22:28:15.872661 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.872610    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f440cc-1942-4ccd-92d6-936b1c59ac30" containerName="payload-pre-processing"
Oct 30 22:28:15.872661 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.872631    4553 cpu_manager.go:414] "containerMap: removing container" podUID="6bbfe405-12a5-4eae-851a-8b2fc99947a0" containerName="maas-api"
Oct 30 22:28:15.872661 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.872636    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bbfe405-12a5-4eae-851a-8b2fc99947a0" containerName="maas-api"
Oct 30 22:28:15.872661 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.872648    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a3a2bd94-6485-452d-98da-b656035143cb" containerName="istio-proxy"
Oct 30 22:28:15.872661 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.872654    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a2bd94-6485-452d-98da-b656035143cb" containerName="istio-proxy"
Oct 30 22:28:15.872814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.872742    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f440cc-1942-4ccd-92d6-936b1c59ac30" containerName="payload-pre-processing"
Oct 30 22:28:15.872814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.872752    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bbfe405-12a5-4eae-851a-8b2fc99947a0" containerName="maas-api"
Oct 30 22:28:15.872814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.872758    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a2bd94-6485-452d-98da-b656035143cb" containerName="istio-proxy"
Oct 30 22:28:15.877399 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.877380    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:15.946113 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.946075    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-maas-api-tls\") pod \"maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:15.946306 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.946127    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a72b7e51-d030-4462-824b-a6ddbf89ea26-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:15.946306 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.946173    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfnbx\" (UniqueName: \"kubernetes.io/projected/a72b7e51-d030-4462-824b-a6ddbf89ea26-kube-api-access-dfnbx\") pod \"maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:15.946502 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.946475    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-metrics-certs\") pod \"maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:15.981248 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.981201    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-42e8a1-serving-cert\""
Oct 30 22:28:15.984296 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:15.984275    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"]
Oct 30 22:28:16.048285 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:16.048248    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a72b7e51-d030-4462-824b-a6ddbf89ea26-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:16.048285 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:16.048288    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfnbx\" (UniqueName: \"kubernetes.io/projected/a72b7e51-d030-4462-824b-a6ddbf89ea26-kube-api-access-dfnbx\") pod \"maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:16.048541 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:16.048481    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-metrics-certs\") pod \"maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:16.048623 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:16.048605    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-maas-api-tls\") pod \"maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:16.048781 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:28:16.048758    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-serving-cert: secret "maas-api-e2e-praxis-a-42e8a1-serving-cert" not found
Oct 30 22:28:16.048854 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:28:16.048843    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-maas-api-tls podName:a72b7e51-d030-4462-824b-a6ddbf89ea26 nodeName:}" failed. No retries permitted until 2026-10-30 22:28:16.548825691 +0000 UTC m=+2747.283188700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-maas-api-tls") pod "maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t" (UID: "a72b7e51-d030-4462-824b-a6ddbf89ea26") : secret "maas-api-e2e-praxis-a-42e8a1-serving-cert" not found
Oct 30 22:28:16.048987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:16.048967    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a72b7e51-d030-4462-824b-a6ddbf89ea26-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:16.051256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:16.051226    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-metrics-certs\") pod \"maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:16.111481 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:16.111452    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfnbx\" (UniqueName: \"kubernetes.io/projected/a72b7e51-d030-4462-824b-a6ddbf89ea26-kube-api-access-dfnbx\") pod \"maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:16.556032 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:16.555995    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-maas-api-tls\") pod \"maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:16.558687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:16.558655    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-maas-api-tls\") pod \"maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:16.792093 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:16.792055    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:17.182116 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:17.182084    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"]
Oct 30 22:28:17.183657 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:28:17.183631    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72b7e51_d030_4462_824b_a6ddbf89ea26.slice/crio-2dba3dd33bc4c3a7091432b82a37e09fee7e4c7c4daad81a8d5df908c42928ec WatchSource:0}: Error finding container 2dba3dd33bc4c3a7091432b82a37e09fee7e4c7c4daad81a8d5df908c42928ec: Status 404 returned error can't find the container with id 2dba3dd33bc4c3a7091432b82a37e09fee7e4c7c4daad81a8d5df908c42928ec
Oct 30 22:28:17.185535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:17.185517    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:28:17.981280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:17.981239    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t" event={"ID":"a72b7e51-d030-4462-824b-a6ddbf89ea26","Type":"ContainerStarted","Data":"2dba3dd33bc4c3a7091432b82a37e09fee7e4c7c4daad81a8d5df908c42928ec"}
Oct 30 22:28:18.991473 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:18.991428    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t" event={"ID":"a72b7e51-d030-4462-824b-a6ddbf89ea26","Type":"ContainerStarted","Data":"5174c4e2924c75dd7b5392521c540f5191010ab957f97889523af6a4aa4970f3"}
Oct 30 22:28:18.991939 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:18.991557    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:19.128984 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:19.128933    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t" podStartSLOduration=3.171405342 podStartE2EDuration="4.128914958s" totalImagesPullingTime="957.509616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:19.125602517 +0000 UTC m=+2749.859965549" watchObservedRunningTime="2026-10-30 22:28:19.128914958 +0000 UTC m=+2749.863278022"
Oct 30 22:28:25.003509 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:25.003480    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:28:28.201780 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:28:28.201742    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:28:36.281730 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:36.281687    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"]
Oct 30 22:28:41.282399 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:41.282340    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb" podUID="455fabb6-e529-4bd2-8247-4d49897f3dce" containerName="payload-pre-processing" containerID="cri-o://f49c56dabc118b22d990f0bc7173676bbbbb520be434b436337df0f6a8617d05" gracePeriod=25
Oct 30 22:28:41.555271 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:41.555229    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"
Oct 30 22:28:41.649014 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:41.648970    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/455fabb6-e529-4bd2-8247-4d49897f3dce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455fabb6-e529-4bd2-8247-4d49897f3dce-plugins-config-volume\") pod \"455fabb6-e529-4bd2-8247-4d49897f3dce\" (UID: \"455fabb6-e529-4bd2-8247-4d49897f3dce\") "
Oct 30 22:28:41.649258 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:41.649240    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/455fabb6-e529-4bd2-8247-4d49897f3dce-kube-api-access-s6j92\" (UniqueName: \"kubernetes.io/projected/455fabb6-e529-4bd2-8247-4d49897f3dce-kube-api-access-s6j92\") pod \"455fabb6-e529-4bd2-8247-4d49897f3dce\" (UID: \"455fabb6-e529-4bd2-8247-4d49897f3dce\") "
Oct 30 22:28:41.649394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:41.649371    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455fabb6-e529-4bd2-8247-4d49897f3dce-plugins-config-volume" pod "455fabb6-e529-4bd2-8247-4d49897f3dce" (UID: "455fabb6-e529-4bd2-8247-4d49897f3dce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:41.649776 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:41.649753    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455fabb6-e529-4bd2-8247-4d49897f3dce-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:41.651505 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:41.651486    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455fabb6-e529-4bd2-8247-4d49897f3dce-kube-api-access-s6j92" pod "455fabb6-e529-4bd2-8247-4d49897f3dce" (UID: "455fabb6-e529-4bd2-8247-4d49897f3dce"). InnerVolumeSpecName "kube-api-access-s6j92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:41.751655 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:41.751614    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6j92\" (UniqueName: \"kubernetes.io/projected/455fabb6-e529-4bd2-8247-4d49897f3dce-kube-api-access-s6j92\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:42.165315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:42.165276    4553 generic.go:356] "Generic (PLEG): container finished" podID="455fabb6-e529-4bd2-8247-4d49897f3dce" containerID="f49c56dabc118b22d990f0bc7173676bbbbb520be434b436337df0f6a8617d05" exitCode=0
Oct 30 22:28:42.165512 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:42.165335    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"
Oct 30 22:28:42.165512 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:42.165349    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb" event={"ID":"455fabb6-e529-4bd2-8247-4d49897f3dce","Type":"ContainerDied","Data":"f49c56dabc118b22d990f0bc7173676bbbbb520be434b436337df0f6a8617d05"}
Oct 30 22:28:42.165512 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:42.165411    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb" event={"ID":"455fabb6-e529-4bd2-8247-4d49897f3dce","Type":"ContainerDied","Data":"30070c39949815e57a871077f359112813010587182f0d1c2a98f329af8f39d1"}
Oct 30 22:28:42.165512 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:42.165434    4553 scope.go:122] "RemoveContainer" containerID="f49c56dabc118b22d990f0bc7173676bbbbb520be434b436337df0f6a8617d05"
Oct 30 22:28:42.176104 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:42.176085    4553 scope.go:122] "RemoveContainer" containerID="f49c56dabc118b22d990f0bc7173676bbbbb520be434b436337df0f6a8617d05"
Oct 30 22:28:42.177625 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:28:42.177594    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49c56dabc118b22d990f0bc7173676bbbbb520be434b436337df0f6a8617d05\": container with ID starting with f49c56dabc118b22d990f0bc7173676bbbbb520be434b436337df0f6a8617d05 not found: ID does not exist" containerID="f49c56dabc118b22d990f0bc7173676bbbbb520be434b436337df0f6a8617d05"
Oct 30 22:28:42.177698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:42.177637    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49c56dabc118b22d990f0bc7173676bbbbb520be434b436337df0f6a8617d05"} err="failed to get container status \"f49c56dabc118b22d990f0bc7173676bbbbb520be434b436337df0f6a8617d05\": rpc error: code = NotFound desc = could not find container \"f49c56dabc118b22d990f0bc7173676bbbbb520be434b436337df0f6a8617d05\": container with ID starting with f49c56dabc118b22d990f0bc7173676bbbbb520be434b436337df0f6a8617d05 not found: ID does not exist"
Oct 30 22:28:42.335868 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:42.335834    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"]
Oct 30 22:28:42.340762 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:42.340736    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5b1e39-649496847-gfkmb"]
Oct 30 22:28:43.393522 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:43.393493    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="455fabb6-e529-4bd2-8247-4d49897f3dce" path="/var/lib/kubelet/pods/455fabb6-e529-4bd2-8247-4d49897f3dce/volumes"
Oct 30 22:28:48.365316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:48.365281    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"]
Oct 30 22:28:48.365764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:48.365506    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8" podUID="c02574d3-4aa9-41f0-91c2-759b5d06654e" containerName="istio-proxy" containerID="cri-o://d7cd651a1f822ad298d4f4b74f241701265626c6a01757e4fdbbfeb7866e4c91" gracePeriod=30
Oct 30 22:28:53.661343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.661314    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:28:53.807749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.807717    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-envoy\") pod \"c02574d3-4aa9-41f0-91c2-759b5d06654e\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") "
Oct 30 22:28:53.807749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.807758    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-podinfo\") pod \"c02574d3-4aa9-41f0-91c2-759b5d06654e\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") "
Oct 30 22:28:53.807970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.807817    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-credential-socket\") pod \"c02574d3-4aa9-41f0-91c2-759b5d06654e\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") "
Oct 30 22:28:53.808086 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.808063    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c02574d3-4aa9-41f0-91c2-759b5d06654e-kube-api-access-4t8vr\" (UniqueName: \"kubernetes.io/projected/c02574d3-4aa9-41f0-91c2-759b5d06654e-kube-api-access-4t8vr\") pod \"c02574d3-4aa9-41f0-91c2-759b5d06654e\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") "
Oct 30 22:28:53.808201 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.808177    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-credential-socket" pod "c02574d3-4aa9-41f0-91c2-759b5d06654e" (UID: "c02574d3-4aa9-41f0-91c2-759b5d06654e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:53.808274 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.808181    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-workload-socket\") pod \"c02574d3-4aa9-41f0-91c2-759b5d06654e\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") "
Oct 30 22:28:53.808274 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.808258    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-token\" (UniqueName: \"kubernetes.io/projected/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-token\") pod \"c02574d3-4aa9-41f0-91c2-759b5d06654e\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") "
Oct 30 22:28:53.808384 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.808346    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-workload-certs\") pod \"c02574d3-4aa9-41f0-91c2-759b5d06654e\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") "
Oct 30 22:28:53.808384 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.808376    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c02574d3-4aa9-41f0-91c2-759b5d06654e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02574d3-4aa9-41f0-91c2-759b5d06654e-istiod-ca-cert\") pod \"c02574d3-4aa9-41f0-91c2-759b5d06654e\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") "
Oct 30 22:28:53.808491 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.808383    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-workload-socket" pod "c02574d3-4aa9-41f0-91c2-759b5d06654e" (UID: "c02574d3-4aa9-41f0-91c2-759b5d06654e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:53.808491 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.808477    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-data\") pod \"c02574d3-4aa9-41f0-91c2-759b5d06654e\" (UID: \"c02574d3-4aa9-41f0-91c2-759b5d06654e\") "
Oct 30 22:28:53.808673 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.808651    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-workload-certs" pod "c02574d3-4aa9-41f0-91c2-759b5d06654e" (UID: "c02574d3-4aa9-41f0-91c2-759b5d06654e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:53.808750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.808721    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c02574d3-4aa9-41f0-91c2-759b5d06654e-istiod-ca-cert" pod "c02574d3-4aa9-41f0-91c2-759b5d06654e" (UID: "c02574d3-4aa9-41f0-91c2-759b5d06654e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:53.808750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.808739    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-data" pod "c02574d3-4aa9-41f0-91c2-759b5d06654e" (UID: "c02574d3-4aa9-41f0-91c2-759b5d06654e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:53.809122 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.809065    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:53.809122 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.809083    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:53.809122 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.809093    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:53.809122 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.809102    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c02574d3-4aa9-41f0-91c2-759b5d06654e-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:53.809122 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.809110    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:53.810530 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.810505    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02574d3-4aa9-41f0-91c2-759b5d06654e-kube-api-access-4t8vr" pod "c02574d3-4aa9-41f0-91c2-759b5d06654e" (UID: "c02574d3-4aa9-41f0-91c2-759b5d06654e"). InnerVolumeSpecName "kube-api-access-4t8vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:53.810663 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.810642    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-podinfo" pod "c02574d3-4aa9-41f0-91c2-759b5d06654e" (UID: "c02574d3-4aa9-41f0-91c2-759b5d06654e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:53.810737 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.810711    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-token" pod "c02574d3-4aa9-41f0-91c2-759b5d06654e" (UID: "c02574d3-4aa9-41f0-91c2-759b5d06654e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:53.810737 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.810722    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-envoy" pod "c02574d3-4aa9-41f0-91c2-759b5d06654e" (UID: "c02574d3-4aa9-41f0-91c2-759b5d06654e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:53.910672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.910578    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t8vr\" (UniqueName: \"kubernetes.io/projected/c02574d3-4aa9-41f0-91c2-759b5d06654e-kube-api-access-4t8vr\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:53.910672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.910610    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:53.910672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.910620    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:53.910672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:53.910630    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c02574d3-4aa9-41f0-91c2-759b5d06654e-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:28:54.250198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:54.250072    4553 generic.go:356] "Generic (PLEG): container finished" podID="c02574d3-4aa9-41f0-91c2-759b5d06654e" containerID="d7cd651a1f822ad298d4f4b74f241701265626c6a01757e4fdbbfeb7866e4c91" exitCode=0
Oct 30 22:28:54.250198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:54.250187    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"
Oct 30 22:28:54.250198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:54.250185    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8" event={"ID":"c02574d3-4aa9-41f0-91c2-759b5d06654e","Type":"ContainerDied","Data":"d7cd651a1f822ad298d4f4b74f241701265626c6a01757e4fdbbfeb7866e4c91"}
Oct 30 22:28:54.250438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:54.250281    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8" event={"ID":"c02574d3-4aa9-41f0-91c2-759b5d06654e","Type":"ContainerDied","Data":"e28c3185a787de440b304bf6f248d811678c150d8a3c645b3042504944c03342"}
Oct 30 22:28:54.250438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:54.250313    4553 scope.go:122] "RemoveContainer" containerID="d7cd651a1f822ad298d4f4b74f241701265626c6a01757e4fdbbfeb7866e4c91"
Oct 30 22:28:54.262252 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:54.262229    4553 scope.go:122] "RemoveContainer" containerID="51ce6e237d8279cf53a6872a6d90773d0eb5b9055f1b1c9805452dfaf6614eec"
Oct 30 22:28:54.272239 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:54.272217    4553 scope.go:122] "RemoveContainer" containerID="d7cd651a1f822ad298d4f4b74f241701265626c6a01757e4fdbbfeb7866e4c91"
Oct 30 22:28:54.272571 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:28:54.272548    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7cd651a1f822ad298d4f4b74f241701265626c6a01757e4fdbbfeb7866e4c91\": container with ID starting with d7cd651a1f822ad298d4f4b74f241701265626c6a01757e4fdbbfeb7866e4c91 not found: ID does not exist" containerID="d7cd651a1f822ad298d4f4b74f241701265626c6a01757e4fdbbfeb7866e4c91"
Oct 30 22:28:54.272661 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:54.272581    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7cd651a1f822ad298d4f4b74f241701265626c6a01757e4fdbbfeb7866e4c91"} err="failed to get container status \"d7cd651a1f822ad298d4f4b74f241701265626c6a01757e4fdbbfeb7866e4c91\": rpc error: code = NotFound desc = could not find container \"d7cd651a1f822ad298d4f4b74f241701265626c6a01757e4fdbbfeb7866e4c91\": container with ID starting with d7cd651a1f822ad298d4f4b74f241701265626c6a01757e4fdbbfeb7866e4c91 not found: ID does not exist"
Oct 30 22:28:54.272661 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:54.272600    4553 scope.go:122] "RemoveContainer" containerID="51ce6e237d8279cf53a6872a6d90773d0eb5b9055f1b1c9805452dfaf6614eec"
Oct 30 22:28:54.272888 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:28:54.272866    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ce6e237d8279cf53a6872a6d90773d0eb5b9055f1b1c9805452dfaf6614eec\": container with ID starting with 51ce6e237d8279cf53a6872a6d90773d0eb5b9055f1b1c9805452dfaf6614eec not found: ID does not exist" containerID="51ce6e237d8279cf53a6872a6d90773d0eb5b9055f1b1c9805452dfaf6614eec"
Oct 30 22:28:54.272934 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:54.272896    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ce6e237d8279cf53a6872a6d90773d0eb5b9055f1b1c9805452dfaf6614eec"} err="failed to get container status \"51ce6e237d8279cf53a6872a6d90773d0eb5b9055f1b1c9805452dfaf6614eec\": rpc error: code = NotFound desc = could not find container \"51ce6e237d8279cf53a6872a6d90773d0eb5b9055f1b1c9805452dfaf6614eec\": container with ID starting with 51ce6e237d8279cf53a6872a6d90773d0eb5b9055f1b1c9805452dfaf6614eec not found: ID does not exist"
Oct 30 22:28:54.417706 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:54.417669    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"]
Oct 30 22:28:54.468442 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:54.468410    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-5b1e39-openshift-default-5c985bc7c-gsvx8"]
Oct 30 22:28:55.394797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:28:55.394755    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c02574d3-4aa9-41f0-91c2-759b5d06654e" path="/var/lib/kubelet/pods/c02574d3-4aa9-41f0-91c2-759b5d06654e/volumes"
Oct 30 22:28:58.375162 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:28:58.368030    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:29:20.477323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.477286    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"]
Oct 30 22:29:20.477886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.477873    4553 cpu_manager.go:414] "containerMap: removing container" podUID="c02574d3-4aa9-41f0-91c2-759b5d06654e" containerName="istio-proxy"
Oct 30 22:29:20.477932 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.477887    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02574d3-4aa9-41f0-91c2-759b5d06654e" containerName="istio-proxy"
Oct 30 22:29:20.477932 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.477926    4553 cpu_manager.go:414] "containerMap: removing container" podUID="455fabb6-e529-4bd2-8247-4d49897f3dce" containerName="payload-pre-processing"
Oct 30 22:29:20.478002 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.477931    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455fabb6-e529-4bd2-8247-4d49897f3dce" containerName="payload-pre-processing"
Oct 30 22:29:20.478040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.478031    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="c02574d3-4aa9-41f0-91c2-759b5d06654e" containerName="istio-proxy"
Oct 30 22:29:20.478093 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.478043    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="455fabb6-e529-4bd2-8247-4d49897f3dce" containerName="payload-pre-processing"
Oct 30 22:29:20.481672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.481655    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.528220 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.528180    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-token\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.528587 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.528555    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnm85\" (UniqueName: \"kubernetes.io/projected/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-kube-api-access-hnm85\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.528818 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.528801    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-credential-socket\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.529064 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.529046    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-podinfo\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.529344 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.529328    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istiod-ca-cert\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.531050 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.531030    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-envoy\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.531774 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.531743    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-workload-socket\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.531979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.531962    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-data\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.532190 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.532160    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-workload-certs\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.534753 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.534732    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-3bbd9b-openshift-default-dockercfg-zb26z\""
Oct 30 22:29:20.591556 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.591523    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"]
Oct 30 22:29:20.634301 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.634262    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-envoy\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.634517 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.634325    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-workload-socket\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.634517 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.634395    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-data\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.634517 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.634452    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-workload-certs\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.634517 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.634494    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-token\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.634749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.634529    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnm85\" (UniqueName: \"kubernetes.io/projected/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-kube-api-access-hnm85\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.634749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.634601    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-credential-socket\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.634749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.634675    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-podinfo\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.634749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.634733    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-workload-socket\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.634967 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.634798    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istiod-ca-cert\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.634967 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.634865    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-data\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.634967 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.634798    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-workload-certs\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.635214 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.635010    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-credential-socket\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.635556 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.635490    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istiod-ca-cert\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.637100 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.637021    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-envoy\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.637360 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.637340    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-podinfo\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.704011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.703968    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-token\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.704181 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.704081    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnm85\" (UniqueName: \"kubernetes.io/projected/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-kube-api-access-hnm85\") pod \"e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") " pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:20.796405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:20.796307    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:21.247873 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:21.247845    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"]
Oct 30 22:29:21.248872 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:29:21.248849    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8af1bfa_6dda_45a6_9435_1ec9b4afd530.slice/crio-a05f1a01bc0c1800cf2ed71674b9d06e04d40d12603874dd3963aef2b873d432 WatchSource:0}: Error finding container a05f1a01bc0c1800cf2ed71674b9d06e04d40d12603874dd3963aef2b873d432: Status 404 returned error can't find the container with id a05f1a01bc0c1800cf2ed71674b9d06e04d40d12603874dd3963aef2b873d432
Oct 30 22:29:21.251376 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:21.251345    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:29:21.251464 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:21.251410    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:29:21.251464 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:21.251444    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:29:21.453811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:21.453775    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44" event={"ID":"f8af1bfa-6dda-45a6-9435-1ec9b4afd530","Type":"ContainerStarted","Data":"53f25c23bcbec20b6c880861c6745417f308881364a7bd824e1d48fffac04ab4"}
Oct 30 22:29:21.453991 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:21.453838    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44" event={"ID":"f8af1bfa-6dda-45a6-9435-1ec9b4afd530","Type":"ContainerStarted","Data":"a05f1a01bc0c1800cf2ed71674b9d06e04d40d12603874dd3963aef2b873d432"}
Oct 30 22:29:21.576350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:21.576243    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44" podStartSLOduration=1.576227203 podStartE2EDuration="1.576227203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:21.575266683 +0000 UTC m=+2812.309629715" watchObservedRunningTime="2026-10-30 22:29:21.576227203 +0000 UTC m=+2812.310590233"
Oct 30 22:29:21.797320 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:21.797282    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:22.802922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:22.802837    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:23.467220 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:23.467185    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:23.468328 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:23.468308    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:29:28.098859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:28.098813    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"]
Oct 30 22:29:28.106159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:28.105718    4553 cpu_manager.go:414] "containerMap: removing container" podUID="c02574d3-4aa9-41f0-91c2-759b5d06654e" containerName="istio-proxy"
Oct 30 22:29:28.106380 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:28.106361    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c02574d3-4aa9-41f0-91c2-759b5d06654e" containerName="istio-proxy"
Oct 30 22:29:28.106678 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:28.106665    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="c02574d3-4aa9-41f0-91c2-759b5d06654e" containerName="istio-proxy"
Oct 30 22:29:28.123453 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:28.123419    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"
Oct 30 22:29:28.170282 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:28.170250    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"]
Oct 30 22:29:28.175326 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:28.175287    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5\" (UID: \"f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"
Oct 30 22:29:28.189167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:28.175676    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54sqw\" (UniqueName: \"kubernetes.io/projected/f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b-kube-api-access-54sqw\") pod \"payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5\" (UID: \"f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"
Oct 30 22:29:28.244352 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:28.201503    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-3bbd9b\""
Oct 30 22:29:28.244610 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:28.202361    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-3bbd9b-dockercfg-wgp4k\""
Oct 30 22:29:28.372846 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:28.372754    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5\" (UID: \"f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"
Oct 30 22:29:28.400309 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:28.400272    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54sqw\" (UniqueName: \"kubernetes.io/projected/f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b-kube-api-access-54sqw\") pod \"payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5\" (UID: \"f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"
Oct 30 22:29:28.426364 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:28.373958    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5\" (UID: \"f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"
Oct 30 22:29:28.623227 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:28.593257    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54sqw\" (UniqueName: \"kubernetes.io/projected/f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b-kube-api-access-54sqw\") pod \"payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5\" (UID: \"f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"
Oct 30 22:29:28.705956 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:29:28.705913    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:29:28.779599 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:28.779562    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"
Oct 30 22:29:29.624695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:29.624666    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"]
Oct 30 22:29:29.625361 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:29:29.625335    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81bb7a1_33d0_4a2b_9a42_067f2bf3ba2b.slice/crio-2623b607564ea4e9a3fcdc5392458a248242e099bbda13f492e1e97c2e1c5a7a WatchSource:0}: Error finding container 2623b607564ea4e9a3fcdc5392458a248242e099bbda13f492e1e97c2e1c5a7a: Status 404 returned error can't find the container with id 2623b607564ea4e9a3fcdc5392458a248242e099bbda13f492e1e97c2e1c5a7a
Oct 30 22:29:29.715634 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:29.715601    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5" event={"ID":"f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b","Type":"ContainerStarted","Data":"2623b607564ea4e9a3fcdc5392458a248242e099bbda13f492e1e97c2e1c5a7a"}
Oct 30 22:29:30.724638 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:30.724598    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5" event={"ID":"f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b","Type":"ContainerStarted","Data":"8849ff46b633c1f9948870fe9185cb1456095f3fbdc73144d8a2d721f56208a5"}
Oct 30 22:29:30.725200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:30.724820    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"
Oct 30 22:29:30.845528 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:30.845453    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5" podStartSLOduration=2.845434417 podStartE2EDuration="2.845434417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:30.840109539 +0000 UTC m=+2821.574472573" watchObservedRunningTime="2026-10-30 22:29:30.845434417 +0000 UTC m=+2821.579797454"
Oct 30 22:29:41.737101 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:29:41.737065    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"
Oct 30 22:29:58.203298 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:29:58.203249    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:30:00.213677 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:00.213639    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"]
Oct 30 22:30:00.218082 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:00.218056    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"
Oct 30 22:30:00.327232 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:00.327202    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:30:00.396921 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:00.396873    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbdjv\" (UniqueName: \"kubernetes.io/projected/9e727db7-7a5b-445d-9b6f-75d186911229-kube-api-access-lbdjv\") pod \"maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8\" (UID: \"9e727db7-7a5b-445d-9b6f-75d186911229\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"
Oct 30 22:30:00.397110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:00.397089    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e727db7-7a5b-445d-9b6f-75d186911229-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8\" (UID: \"9e727db7-7a5b-445d-9b6f-75d186911229\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"
Oct 30 22:30:00.433097 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:00.433062    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"]
Oct 30 22:30:00.498427 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:00.498340    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e727db7-7a5b-445d-9b6f-75d186911229-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8\" (UID: \"9e727db7-7a5b-445d-9b6f-75d186911229\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"
Oct 30 22:30:00.498583 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:00.498443    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbdjv\" (UniqueName: \"kubernetes.io/projected/9e727db7-7a5b-445d-9b6f-75d186911229-kube-api-access-lbdjv\") pod \"maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8\" (UID: \"9e727db7-7a5b-445d-9b6f-75d186911229\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"
Oct 30 22:30:00.499064 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:00.499045    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e727db7-7a5b-445d-9b6f-75d186911229-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8\" (UID: \"9e727db7-7a5b-445d-9b6f-75d186911229\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"
Oct 30 22:30:00.556032 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:00.555995    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbdjv\" (UniqueName: \"kubernetes.io/projected/9e727db7-7a5b-445d-9b6f-75d186911229-kube-api-access-lbdjv\") pod \"maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8\" (UID: \"9e727db7-7a5b-445d-9b6f-75d186911229\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"
Oct 30 22:30:00.832691 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:00.832663    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"
Oct 30 22:30:01.217322 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:01.217292    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"]
Oct 30 22:30:01.218177 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:30:01.218129    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e727db7_7a5b_445d_9b6f_75d186911229.slice/crio-9de02be49b873a681520f446c800485170980f2bbd87539de158eddde3a00261 WatchSource:0}: Error finding container 9de02be49b873a681520f446c800485170980f2bbd87539de158eddde3a00261: Status 404 returned error can't find the container with id 9de02be49b873a681520f446c800485170980f2bbd87539de158eddde3a00261
Oct 30 22:30:01.969675 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:01.969643    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8" event={"ID":"9e727db7-7a5b-445d-9b6f-75d186911229","Type":"ContainerStarted","Data":"f567d0fbf72aeaa239667c38fe8394a88067fd18db05104f6a846a9597642440"}
Oct 30 22:30:01.969873 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:01.969693    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8" event={"ID":"9e727db7-7a5b-445d-9b6f-75d186911229","Type":"ContainerStarted","Data":"9de02be49b873a681520f446c800485170980f2bbd87539de158eddde3a00261"}
Oct 30 22:30:02.086811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:02.086758    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8" podStartSLOduration=2.086744689 podStartE2EDuration="2.086744689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.086189376 +0000 UTC m=+2852.820552408" watchObservedRunningTime="2026-10-30 22:30:02.086744689 +0000 UTC m=+2852.821107716"
Oct 30 22:30:02.978223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:02.978180    4553 generic.go:356] "Generic (PLEG): container finished" podID="9e727db7-7a5b-445d-9b6f-75d186911229" containerID="f567d0fbf72aeaa239667c38fe8394a88067fd18db05104f6a846a9597642440" exitCode=60
Oct 30 22:30:02.978686 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:02.978260    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8" event={"ID":"9e727db7-7a5b-445d-9b6f-75d186911229","Type":"ContainerDied","Data":"f567d0fbf72aeaa239667c38fe8394a88067fd18db05104f6a846a9597642440"}
Oct 30 22:30:02.978686 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:02.978601    4553 scope.go:122] "RemoveContainer" containerID="f567d0fbf72aeaa239667c38fe8394a88067fd18db05104f6a846a9597642440"
Oct 30 22:30:03.986973 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:03.986940    4553 generic.go:356] "Generic (PLEG): container finished" podID="9e727db7-7a5b-445d-9b6f-75d186911229" containerID="d4518d589783427582e217506a7b66aa1ef1655f6c0f229c5752177a26903ff1" exitCode=60
Oct 30 22:30:03.987524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:03.987022    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8" event={"ID":"9e727db7-7a5b-445d-9b6f-75d186911229","Type":"ContainerDied","Data":"d4518d589783427582e217506a7b66aa1ef1655f6c0f229c5752177a26903ff1"}
Oct 30 22:30:03.987524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:03.987086    4553 scope.go:122] "RemoveContainer" containerID="f567d0fbf72aeaa239667c38fe8394a88067fd18db05104f6a846a9597642440"
Oct 30 22:30:03.987524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:03.987441    4553 scope.go:122] "RemoveContainer" containerID="d4518d589783427582e217506a7b66aa1ef1655f6c0f229c5752177a26903ff1"
Oct 30 22:30:03.987826 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:30:03.987802    4553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8_odh-ai-gateway-infra(9e727db7-7a5b-445d-9b6f-75d186911229)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8" podUID="9e727db7-7a5b-445d-9b6f-75d186911229"
Oct 30 22:30:04.464406 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.464363    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"]
Oct 30 22:30:04.469896 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.469857    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.528111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.528075    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"]
Oct 30 22:30:04.574257 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.574229    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b8717e-openshift-default-dockercfg-pqzrg\""
Oct 30 22:30:04.652486 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.652448    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.652486 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.652490    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.652753 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.652514    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.652753 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.652614    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.652753 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.652666    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.652924 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.652769    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.652984 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.652947    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.653042 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.652990    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.653104 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.653075    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htsr4\" (UniqueName: \"kubernetes.io/projected/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-kube-api-access-htsr4\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.754343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.754249    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.754343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.754298    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.754599 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.754391    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.754599 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.754418    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.754599 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.754454    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htsr4\" (UniqueName: \"kubernetes.io/projected/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-kube-api-access-htsr4\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.754599 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.754533    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.754599 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.754562    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.754599 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.754594    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.754940 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.754692    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.754940 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.754724    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.754940 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.754832    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.755093 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.754986    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.755203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.755120    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.755291 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.755266    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.757003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.756972    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.757228 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.757208    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.817250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.817219    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.817650 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.817628    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htsr4\" (UniqueName: \"kubernetes.io/projected/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-kube-api-access-htsr4\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:04.995527 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:04.995498    4553 scope.go:122] "RemoveContainer" containerID="d4518d589783427582e217506a7b66aa1ef1655f6c0f229c5752177a26903ff1"
Oct 30 22:30:04.995995 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:30:04.995907    4553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8_odh-ai-gateway-infra(9e727db7-7a5b-445d-9b6f-75d186911229)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8" podUID="9e727db7-7a5b-445d-9b6f-75d186911229"
Oct 30 22:30:05.088432 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:05.088401    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:05.519039 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:05.519010    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"]
Oct 30 22:30:05.519509 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:30:05.519483    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769f2ed9_d041_4d27_ba11_3a6ad3ebf455.slice/crio-507543d6295075b644531f3abcfea7c57e8e7f084b40fd8a820fea7bc8c8ea4b WatchSource:0}: Error finding container 507543d6295075b644531f3abcfea7c57e8e7f084b40fd8a820fea7bc8c8ea4b: Status 404 returned error can't find the container with id 507543d6295075b644531f3abcfea7c57e8e7f084b40fd8a820fea7bc8c8ea4b
Oct 30 22:30:05.522784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:05.522749    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:30:05.522882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:05.522820    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:30:05.522882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:05.522849    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:30:06.005275 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:06.005234    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68" event={"ID":"769f2ed9-d041-4d27-ba11-3a6ad3ebf455","Type":"ContainerStarted","Data":"a499ae04bc87bd9a0c75b20a624355f584168304cdeba0054274b897c38e8f34"}
Oct 30 22:30:06.005693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:06.005305    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68" event={"ID":"769f2ed9-d041-4d27-ba11-3a6ad3ebf455","Type":"ContainerStarted","Data":"507543d6295075b644531f3abcfea7c57e8e7f084b40fd8a820fea7bc8c8ea4b"}
Oct 30 22:30:06.089047 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:06.089017    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:06.090386 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:06.090364    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:30:06.090453 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:06.090409    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68" podUID="769f2ed9-d041-4d27-ba11-3a6ad3ebf455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 22:30:06.137255 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:06.137198    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68" podStartSLOduration=2.137182389 podStartE2EDuration="2.137182389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:06.133950607 +0000 UTC m=+2856.868313640" watchObservedRunningTime="2026-10-30 22:30:06.137182389 +0000 UTC m=+2856.871545497"
Oct 30 22:30:07.089281 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:07.089244    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:30:07.089872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:07.089840    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68" podUID="769f2ed9-d041-4d27-ba11-3a6ad3ebf455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 22:30:08.097164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:08.095591    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:30:08.097164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:08.095657    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68" podUID="769f2ed9-d041-4d27-ba11-3a6ad3ebf455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 22:30:09.089162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:09.088983    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:30:09.089162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:09.089045    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68" podUID="769f2ed9-d041-4d27-ba11-3a6ad3ebf455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 22:30:10.088743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:10.088707    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:30:10.089154 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:10.088765    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68" podUID="769f2ed9-d041-4d27-ba11-3a6ad3ebf455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 22:30:10.335475 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:10.335429    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"]
Oct 30 22:30:10.435518 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:10.435411    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"]
Oct 30 22:30:11.088960 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:11.088926    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:30:11.089418 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:11.088985    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68" podUID="769f2ed9-d041-4d27-ba11-3a6ad3ebf455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 22:30:12.088881 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:12.088850    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Oct 30 22:30:12.089052 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:12.088905    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68" podUID="769f2ed9-d041-4d27-ba11-3a6ad3ebf455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Oct 30 22:30:13.878676 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:13.878627    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"]
Oct 30 22:30:13.884110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:13.884085    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:13.906407 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:13.906369    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-metrics-certs\") pod \"maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:13.906591 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:13.906518    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:13.906591 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:13.906563    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgntd\" (UniqueName: \"kubernetes.io/projected/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-kube-api-access-xgntd\") pod \"maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:13.906759 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:13.906743    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:13.937075 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:13.937040    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-3bbd9b-serving-cert\""
Oct 30 22:30:13.995451 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:13.995414    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"]
Oct 30 22:30:14.007764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:14.007726    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:14.008556 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:14.008525    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-metrics-certs\") pod \"maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:14.008959 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:14.008940    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:14.009263 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:14.009242    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgntd\" (UniqueName: \"kubernetes.io/projected/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-kube-api-access-xgntd\") pod \"maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:14.009851 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:30:14.009183    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-serving-cert: secret "maas-api-e2e-praxis-b-3bbd9b-serving-cert" not found
Oct 30 22:30:14.010820 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:30:14.010804    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-maas-api-tls podName:d9fe0777-d9c7-476d-bf9c-9cd0a87394e3 nodeName:}" failed. No retries permitted until 2026-10-30 22:30:14.510781939 +0000 UTC m=+2865.245144951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-maas-api-tls") pod "maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m" (UID: "d9fe0777-d9c7-476d-bf9c-9cd0a87394e3") : secret "maas-api-e2e-praxis-b-3bbd9b-serving-cert" not found
Oct 30 22:30:14.011190 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:14.010440    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:14.032085 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:14.031985    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-metrics-certs\") pod \"maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:14.072418 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:14.072375    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgntd\" (UniqueName: \"kubernetes.io/projected/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-kube-api-access-xgntd\") pod \"maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:14.134188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:14.133425    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:30:14.134188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:14.133511    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68" podUID="769f2ed9-d041-4d27-ba11-3a6ad3ebf455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:30:14.518145 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:14.518041    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:14.521215 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:14.521176    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:14.542656 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:14.542626    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:14.543084 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:14.543059    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:14.543904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:14.543882    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:30:14.798752 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:14.798657    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:15.196444 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.196412    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"]
Oct 30 22:30:15.196825 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:30:15.196623    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9fe0777_d9c7_476d_bf9c_9cd0a87394e3.slice/crio-edcf4ab57075f6e6797a8999fc246beb3f6479336ea9ab7c0e66e5b5b7bbfe6d WatchSource:0}: Error finding container edcf4ab57075f6e6797a8999fc246beb3f6479336ea9ab7c0e66e5b5b7bbfe6d: Status 404 returned error can't find the container with id edcf4ab57075f6e6797a8999fc246beb3f6479336ea9ab7c0e66e5b5b7bbfe6d
Oct 30 22:30:15.336630 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.336577    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97" podUID="86123dfa-c061-4101-a32f-9788677cdb14" containerName="payload-processing" containerID="cri-o://456a3b7d5041be643405222909e91e3fa2c4758b5a71eadfff7e4ba679dba2e9" gracePeriod=25
Oct 30 22:30:15.436192 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.436113    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx" podUID="83795218-018c-486a-b209-150f6c424d73" containerName="payload-pre-processing" containerID="cri-o://e2db169a00e290e518134deeb143cb8f4ca9d8ebd416ff7d597386a35038d6ca" gracePeriod=25
Oct 30 22:30:15.646638 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.646605    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"
Oct 30 22:30:15.707652 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.707589    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"
Oct 30 22:30:15.734745 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.734717    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86123dfa-c061-4101-a32f-9788677cdb14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86123dfa-c061-4101-a32f-9788677cdb14-plugins-config-volume\") pod \"86123dfa-c061-4101-a32f-9788677cdb14\" (UID: \"86123dfa-c061-4101-a32f-9788677cdb14\") "
Oct 30 22:30:15.734924 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.734797    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86123dfa-c061-4101-a32f-9788677cdb14-kube-api-access-xrdpx\" (UniqueName: \"kubernetes.io/projected/86123dfa-c061-4101-a32f-9788677cdb14-kube-api-access-xrdpx\") pod \"86123dfa-c061-4101-a32f-9788677cdb14\" (UID: \"86123dfa-c061-4101-a32f-9788677cdb14\") "
Oct 30 22:30:15.735228 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.735194    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86123dfa-c061-4101-a32f-9788677cdb14-plugins-config-volume" pod "86123dfa-c061-4101-a32f-9788677cdb14" (UID: "86123dfa-c061-4101-a32f-9788677cdb14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:15.737144 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.737110    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86123dfa-c061-4101-a32f-9788677cdb14-kube-api-access-xrdpx" pod "86123dfa-c061-4101-a32f-9788677cdb14" (UID: "86123dfa-c061-4101-a32f-9788677cdb14"). InnerVolumeSpecName "kube-api-access-xrdpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:15.836526 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.836498    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83795218-018c-486a-b209-150f6c424d73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83795218-018c-486a-b209-150f6c424d73-plugins-config-volume\") pod \"83795218-018c-486a-b209-150f6c424d73\" (UID: \"83795218-018c-486a-b209-150f6c424d73\") "
Oct 30 22:30:15.836701 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.836562    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83795218-018c-486a-b209-150f6c424d73-kube-api-access-69hgl\" (UniqueName: \"kubernetes.io/projected/83795218-018c-486a-b209-150f6c424d73-kube-api-access-69hgl\") pod \"83795218-018c-486a-b209-150f6c424d73\" (UID: \"83795218-018c-486a-b209-150f6c424d73\") "
Oct 30 22:30:15.836930 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.836908    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83795218-018c-486a-b209-150f6c424d73-plugins-config-volume" pod "83795218-018c-486a-b209-150f6c424d73" (UID: "83795218-018c-486a-b209-150f6c424d73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:15.837195 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.837169    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83795218-018c-486a-b209-150f6c424d73-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:15.837195 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.837197    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86123dfa-c061-4101-a32f-9788677cdb14-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:15.837371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.837207    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrdpx\" (UniqueName: \"kubernetes.io/projected/86123dfa-c061-4101-a32f-9788677cdb14-kube-api-access-xrdpx\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:15.839003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.838976    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83795218-018c-486a-b209-150f6c424d73-kube-api-access-69hgl" pod "83795218-018c-486a-b209-150f6c424d73" (UID: "83795218-018c-486a-b209-150f6c424d73"). InnerVolumeSpecName "kube-api-access-69hgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:15.938522 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:15.938464    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69hgl\" (UniqueName: \"kubernetes.io/projected/83795218-018c-486a-b209-150f6c424d73-kube-api-access-69hgl\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:16.154199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.154164    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m" event={"ID":"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3","Type":"ContainerStarted","Data":"edcf4ab57075f6e6797a8999fc246beb3f6479336ea9ab7c0e66e5b5b7bbfe6d"}
Oct 30 22:30:16.155672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.155646    4553 generic.go:356] "Generic (PLEG): container finished" podID="83795218-018c-486a-b209-150f6c424d73" containerID="e2db169a00e290e518134deeb143cb8f4ca9d8ebd416ff7d597386a35038d6ca" exitCode=0
Oct 30 22:30:16.155828 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.155694    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"
Oct 30 22:30:16.155828 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.155729    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx" event={"ID":"83795218-018c-486a-b209-150f6c424d73","Type":"ContainerDied","Data":"e2db169a00e290e518134deeb143cb8f4ca9d8ebd416ff7d597386a35038d6ca"}
Oct 30 22:30:16.155828 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.155784    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx" event={"ID":"83795218-018c-486a-b209-150f6c424d73","Type":"ContainerDied","Data":"da9648bf2832aade7266446839b0c6b4ed7aebc98310a5cc070ee985906f9df0"}
Oct 30 22:30:16.155828 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.155815    4553 scope.go:122] "RemoveContainer" containerID="e2db169a00e290e518134deeb143cb8f4ca9d8ebd416ff7d597386a35038d6ca"
Oct 30 22:30:16.157397 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.157370    4553 generic.go:356] "Generic (PLEG): container finished" podID="86123dfa-c061-4101-a32f-9788677cdb14" containerID="456a3b7d5041be643405222909e91e3fa2c4758b5a71eadfff7e4ba679dba2e9" exitCode=0
Oct 30 22:30:16.157545 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.157417    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"
Oct 30 22:30:16.157545 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.157457    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97" event={"ID":"86123dfa-c061-4101-a32f-9788677cdb14","Type":"ContainerDied","Data":"456a3b7d5041be643405222909e91e3fa2c4758b5a71eadfff7e4ba679dba2e9"}
Oct 30 22:30:16.157696 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.157669    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97" event={"ID":"86123dfa-c061-4101-a32f-9788677cdb14","Type":"ContainerDied","Data":"6cb35ad31affa268cfec5c58243ed27b169ad11bc7a777e78b84f3045fc9ca97"}
Oct 30 22:30:16.169482 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.169461    4553 scope.go:122] "RemoveContainer" containerID="e2db169a00e290e518134deeb143cb8f4ca9d8ebd416ff7d597386a35038d6ca"
Oct 30 22:30:16.169908 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:30:16.169882    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2db169a00e290e518134deeb143cb8f4ca9d8ebd416ff7d597386a35038d6ca\": container with ID starting with e2db169a00e290e518134deeb143cb8f4ca9d8ebd416ff7d597386a35038d6ca not found: ID does not exist" containerID="e2db169a00e290e518134deeb143cb8f4ca9d8ebd416ff7d597386a35038d6ca"
Oct 30 22:30:16.169999 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.169920    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2db169a00e290e518134deeb143cb8f4ca9d8ebd416ff7d597386a35038d6ca"} err="failed to get container status \"e2db169a00e290e518134deeb143cb8f4ca9d8ebd416ff7d597386a35038d6ca\": rpc error: code = NotFound desc = could not find container \"e2db169a00e290e518134deeb143cb8f4ca9d8ebd416ff7d597386a35038d6ca\": container with ID starting with e2db169a00e290e518134deeb143cb8f4ca9d8ebd416ff7d597386a35038d6ca not found: ID does not exist"
Oct 30 22:30:16.169999 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.169948    4553 scope.go:122] "RemoveContainer" containerID="456a3b7d5041be643405222909e91e3fa2c4758b5a71eadfff7e4ba679dba2e9"
Oct 30 22:30:16.186211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.186189    4553 scope.go:122] "RemoveContainer" containerID="456a3b7d5041be643405222909e91e3fa2c4758b5a71eadfff7e4ba679dba2e9"
Oct 30 22:30:16.186591 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:30:16.186571    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456a3b7d5041be643405222909e91e3fa2c4758b5a71eadfff7e4ba679dba2e9\": container with ID starting with 456a3b7d5041be643405222909e91e3fa2c4758b5a71eadfff7e4ba679dba2e9 not found: ID does not exist" containerID="456a3b7d5041be643405222909e91e3fa2c4758b5a71eadfff7e4ba679dba2e9"
Oct 30 22:30:16.186656 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.186603    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456a3b7d5041be643405222909e91e3fa2c4758b5a71eadfff7e4ba679dba2e9"} err="failed to get container status \"456a3b7d5041be643405222909e91e3fa2c4758b5a71eadfff7e4ba679dba2e9\": rpc error: code = NotFound desc = could not find container \"456a3b7d5041be643405222909e91e3fa2c4758b5a71eadfff7e4ba679dba2e9\": container with ID starting with 456a3b7d5041be643405222909e91e3fa2c4758b5a71eadfff7e4ba679dba2e9 not found: ID does not exist"
Oct 30 22:30:16.324589 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.324548    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"]
Oct 30 22:30:16.358055 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.358010    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-a853fe-9ffb88c9f-wnthx"]
Oct 30 22:30:16.386413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.386316    4553 scope.go:122] "RemoveContainer" containerID="d4518d589783427582e217506a7b66aa1ef1655f6c0f229c5752177a26903ff1"
Oct 30 22:30:16.438170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.438110    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"]
Oct 30 22:30:16.442191 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:16.442162    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-a853fe-86b8f5c447-n6g97"]
Oct 30 22:30:17.168781 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:17.168742    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8" event={"ID":"9e727db7-7a5b-445d-9b6f-75d186911229","Type":"ContainerStarted","Data":"f6a31b497443bfb41ee43884f910d054ea90a7c3247d6c7829e744ba4ecb3e94"}
Oct 30 22:30:17.171818 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:17.171779    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m" event={"ID":"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3","Type":"ContainerStarted","Data":"af514ececf9d1bbfcf52a4f07b567b57fa9302b9ea84579037864b50871517a9"}
Oct 30 22:30:17.172000 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:17.171911    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:17.394862 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:17.394826    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83795218-018c-486a-b209-150f6c424d73" path="/var/lib/kubelet/pods/83795218-018c-486a-b209-150f6c424d73/volumes"
Oct 30 22:30:17.395356 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:17.395339    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86123dfa-c061-4101-a32f-9788677cdb14" path="/var/lib/kubelet/pods/86123dfa-c061-4101-a32f-9788677cdb14/volumes"
Oct 30 22:30:17.414632 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:17.414579    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m" podStartSLOduration=3.487376427 podStartE2EDuration="4.414565812s" totalImagesPullingTime="927.189385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:17.412800545 +0000 UTC m=+2868.147163575" watchObservedRunningTime="2026-10-30 22:30:17.414565812 +0000 UTC m=+2868.148928843"
Oct 30 22:30:18.183088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:18.183057    4553 generic.go:356] "Generic (PLEG): container finished" podID="9e727db7-7a5b-445d-9b6f-75d186911229" containerID="f6a31b497443bfb41ee43884f910d054ea90a7c3247d6c7829e744ba4ecb3e94" exitCode=60
Oct 30 22:30:18.183288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:18.183125    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8" event={"ID":"9e727db7-7a5b-445d-9b6f-75d186911229","Type":"ContainerDied","Data":"f6a31b497443bfb41ee43884f910d054ea90a7c3247d6c7829e744ba4ecb3e94"}
Oct 30 22:30:18.183288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:18.183210    4553 scope.go:122] "RemoveContainer" containerID="d4518d589783427582e217506a7b66aa1ef1655f6c0f229c5752177a26903ff1"
Oct 30 22:30:18.183784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:18.183764    4553 scope.go:122] "RemoveContainer" containerID="f6a31b497443bfb41ee43884f910d054ea90a7c3247d6c7829e744ba4ecb3e94"
Oct 30 22:30:18.184095 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:30:18.184075    4553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8_odh-ai-gateway-infra(9e727db7-7a5b-445d-9b6f-75d186911229)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8" podUID="9e727db7-7a5b-445d-9b6f-75d186911229"
Oct 30 22:30:18.316522 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:18.316492    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"]
Oct 30 22:30:19.349007 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:19.348977    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"
Oct 30 22:30:19.490117 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:19.490021    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e727db7-7a5b-445d-9b6f-75d186911229-kube-api-access-lbdjv\" (UniqueName: \"kubernetes.io/projected/9e727db7-7a5b-445d-9b6f-75d186911229-kube-api-access-lbdjv\") pod \"9e727db7-7a5b-445d-9b6f-75d186911229\" (UID: \"9e727db7-7a5b-445d-9b6f-75d186911229\") "
Oct 30 22:30:19.490557 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:19.490512    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e727db7-7a5b-445d-9b6f-75d186911229-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e727db7-7a5b-445d-9b6f-75d186911229-maas-api-service-ca\") pod \"9e727db7-7a5b-445d-9b6f-75d186911229\" (UID: \"9e727db7-7a5b-445d-9b6f-75d186911229\") "
Oct 30 22:30:19.491056 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:19.491026    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e727db7-7a5b-445d-9b6f-75d186911229-maas-api-service-ca" pod "9e727db7-7a5b-445d-9b6f-75d186911229" (UID: "9e727db7-7a5b-445d-9b6f-75d186911229"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.491430 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:19.491408    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e727db7-7a5b-445d-9b6f-75d186911229-maas-api-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.493259 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:19.493218    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e727db7-7a5b-445d-9b6f-75d186911229-kube-api-access-lbdjv" pod "9e727db7-7a5b-445d-9b6f-75d186911229" (UID: "9e727db7-7a5b-445d-9b6f-75d186911229"). InnerVolumeSpecName "kube-api-access-lbdjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.592972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:19.592931    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbdjv\" (UniqueName: \"kubernetes.io/projected/9e727db7-7a5b-445d-9b6f-75d186911229-kube-api-access-lbdjv\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.200739 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:20.200705    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"
Oct 30 22:30:20.200970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:20.200742    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8" event={"ID":"9e727db7-7a5b-445d-9b6f-75d186911229","Type":"ContainerDied","Data":"9de02be49b873a681520f446c800485170980f2bbd87539de158eddde3a00261"}
Oct 30 22:30:20.200970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:20.200809    4553 scope.go:122] "RemoveContainer" containerID="f6a31b497443bfb41ee43884f910d054ea90a7c3247d6c7829e744ba4ecb3e94"
Oct 30 22:30:20.368947 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:20.368906    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"]
Oct 30 22:30:20.419038 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:20.418993    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-a853fe-29889990-8nkg8"]
Oct 30 22:30:21.394471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:21.394430    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e727db7-7a5b-445d-9b6f-75d186911229" path="/var/lib/kubelet/pods/9e727db7-7a5b-445d-9b6f-75d186911229/volumes"
Oct 30 22:30:23.189621 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.189591    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:30:23.296557 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.296517    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"]
Oct 30 22:30:23.296759 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.296729    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm" podUID="15b7811c-45bc-49ff-a3e2-debaedc77a89" containerName="maas-api" containerID="cri-o://1015b15cd0fc0b4c82d1ae63dd801c20e728d152fa35fcba662e81c62c1a6c8f" gracePeriod=30
Oct 30 22:30:23.593920 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.593894    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:30:23.755784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.755696    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b7811c-45bc-49ff-a3e2-debaedc77a89-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b7811c-45bc-49ff-a3e2-debaedc77a89-openshift-service-ca\") pod \"15b7811c-45bc-49ff-a3e2-debaedc77a89\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") "
Oct 30 22:30:23.755951 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.755931    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-maas-api-tls\") pod \"15b7811c-45bc-49ff-a3e2-debaedc77a89\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") "
Oct 30 22:30:23.756016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.756001    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b7811c-45bc-49ff-a3e2-debaedc77a89-kube-api-access-96f7v\" (UniqueName: \"kubernetes.io/projected/15b7811c-45bc-49ff-a3e2-debaedc77a89-kube-api-access-96f7v\") pod \"15b7811c-45bc-49ff-a3e2-debaedc77a89\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") "
Oct 30 22:30:23.756083 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.756067    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-metrics-certs\") pod \"15b7811c-45bc-49ff-a3e2-debaedc77a89\" (UID: \"15b7811c-45bc-49ff-a3e2-debaedc77a89\") "
Oct 30 22:30:23.756155 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.756111    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b7811c-45bc-49ff-a3e2-debaedc77a89-openshift-service-ca" pod "15b7811c-45bc-49ff-a3e2-debaedc77a89" (UID: "15b7811c-45bc-49ff-a3e2-debaedc77a89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:23.756613 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.756588    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b7811c-45bc-49ff-a3e2-debaedc77a89-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:23.758576 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.758552    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-maas-api-tls" pod "15b7811c-45bc-49ff-a3e2-debaedc77a89" (UID: "15b7811c-45bc-49ff-a3e2-debaedc77a89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:23.758685 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.758604    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-metrics-certs" pod "15b7811c-45bc-49ff-a3e2-debaedc77a89" (UID: "15b7811c-45bc-49ff-a3e2-debaedc77a89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:23.758685 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.758674    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b7811c-45bc-49ff-a3e2-debaedc77a89-kube-api-access-96f7v" pod "15b7811c-45bc-49ff-a3e2-debaedc77a89" (UID: "15b7811c-45bc-49ff-a3e2-debaedc77a89"). InnerVolumeSpecName "kube-api-access-96f7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:23.857799 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.857754    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:23.857799 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.857800    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96f7v\" (UniqueName: \"kubernetes.io/projected/15b7811c-45bc-49ff-a3e2-debaedc77a89-kube-api-access-96f7v\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:23.858070 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:23.857817    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15b7811c-45bc-49ff-a3e2-debaedc77a89-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:24.270417 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:24.270348    4553 generic.go:356] "Generic (PLEG): container finished" podID="15b7811c-45bc-49ff-a3e2-debaedc77a89" containerID="1015b15cd0fc0b4c82d1ae63dd801c20e728d152fa35fcba662e81c62c1a6c8f" exitCode=0
Oct 30 22:30:24.270417 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:24.270410    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"
Oct 30 22:30:24.270417 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:24.270439    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm" event={"ID":"15b7811c-45bc-49ff-a3e2-debaedc77a89","Type":"ContainerDied","Data":"1015b15cd0fc0b4c82d1ae63dd801c20e728d152fa35fcba662e81c62c1a6c8f"}
Oct 30 22:30:24.271094 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:24.270500    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm" event={"ID":"15b7811c-45bc-49ff-a3e2-debaedc77a89","Type":"ContainerDied","Data":"ead41af205e4801f6060d5361b0f178c4eec0055544ca6d28db6828df11e3892"}
Oct 30 22:30:24.271094 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:24.270530    4553 scope.go:122] "RemoveContainer" containerID="1015b15cd0fc0b4c82d1ae63dd801c20e728d152fa35fcba662e81c62c1a6c8f"
Oct 30 22:30:24.283974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:24.283942    4553 scope.go:122] "RemoveContainer" containerID="1015b15cd0fc0b4c82d1ae63dd801c20e728d152fa35fcba662e81c62c1a6c8f"
Oct 30 22:30:24.284344 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:30:24.284321    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1015b15cd0fc0b4c82d1ae63dd801c20e728d152fa35fcba662e81c62c1a6c8f\": container with ID starting with 1015b15cd0fc0b4c82d1ae63dd801c20e728d152fa35fcba662e81c62c1a6c8f not found: ID does not exist" containerID="1015b15cd0fc0b4c82d1ae63dd801c20e728d152fa35fcba662e81c62c1a6c8f"
Oct 30 22:30:24.284437 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:24.284354    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1015b15cd0fc0b4c82d1ae63dd801c20e728d152fa35fcba662e81c62c1a6c8f"} err="failed to get container status \"1015b15cd0fc0b4c82d1ae63dd801c20e728d152fa35fcba662e81c62c1a6c8f\": rpc error: code = NotFound desc = could not find container \"1015b15cd0fc0b4c82d1ae63dd801c20e728d152fa35fcba662e81c62c1a6c8f\": container with ID starting with 1015b15cd0fc0b4c82d1ae63dd801c20e728d152fa35fcba662e81c62c1a6c8f not found: ID does not exist"
Oct 30 22:30:24.438985 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:24.438942    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"]
Oct 30 22:30:24.489273 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:24.489240    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a853fe-646cfb7ffc-n2lgm"]
Oct 30 22:30:25.394525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:25.394486    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b7811c-45bc-49ff-a3e2-debaedc77a89" path="/var/lib/kubelet/pods/15b7811c-45bc-49ff-a3e2-debaedc77a89/volumes"
Oct 30 22:30:28.203474 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:30:28.203438    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:30:32.479791 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:32.479758    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"]
Oct 30 22:30:32.480412 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:32.479964    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc" podUID="6a77b4ea-e62c-4f91-9264-d037dbe9210e" containerName="istio-proxy" containerID="cri-o://0ec280cf0f4e5a8b58349f718d6b866e3db51dd564e623539534340a8136fc03" gracePeriod=30
Oct 30 22:30:35.219171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.218015    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"]
Oct 30 22:30:35.219171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.218962    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9e727db7-7a5b-445d-9b6f-75d186911229" containerName="cleanup"
Oct 30 22:30:35.219171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.218983    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e727db7-7a5b-445d-9b6f-75d186911229" containerName="cleanup"
Oct 30 22:30:35.219171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.219004    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9e727db7-7a5b-445d-9b6f-75d186911229" containerName="cleanup"
Oct 30 22:30:35.219171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.219011    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e727db7-7a5b-445d-9b6f-75d186911229" containerName="cleanup"
Oct 30 22:30:35.219171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.219032    4553 cpu_manager.go:414] "containerMap: removing container" podUID="86123dfa-c061-4101-a32f-9788677cdb14" containerName="payload-processing"
Oct 30 22:30:35.219171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.219039    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86123dfa-c061-4101-a32f-9788677cdb14" containerName="payload-processing"
Oct 30 22:30:35.219171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.219052    4553 cpu_manager.go:414] "containerMap: removing container" podUID="15b7811c-45bc-49ff-a3e2-debaedc77a89" containerName="maas-api"
Oct 30 22:30:35.219171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.219059    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b7811c-45bc-49ff-a3e2-debaedc77a89" containerName="maas-api"
Oct 30 22:30:35.219171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.219075    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9e727db7-7a5b-445d-9b6f-75d186911229" containerName="cleanup"
Oct 30 22:30:35.219171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.219082    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e727db7-7a5b-445d-9b6f-75d186911229" containerName="cleanup"
Oct 30 22:30:35.219171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.219108    4553 cpu_manager.go:414] "containerMap: removing container" podUID="83795218-018c-486a-b209-150f6c424d73" containerName="payload-pre-processing"
Oct 30 22:30:35.219171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.219116    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83795218-018c-486a-b209-150f6c424d73" containerName="payload-pre-processing"
Oct 30 22:30:35.220303 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.219348    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="83795218-018c-486a-b209-150f6c424d73" containerName="payload-pre-processing"
Oct 30 22:30:35.220303 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.219370    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e727db7-7a5b-445d-9b6f-75d186911229" containerName="cleanup"
Oct 30 22:30:35.220303 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.219379    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e727db7-7a5b-445d-9b6f-75d186911229" containerName="cleanup"
Oct 30 22:30:35.220303 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.219390    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="86123dfa-c061-4101-a32f-9788677cdb14" containerName="payload-processing"
Oct 30 22:30:35.220303 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.219399    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b7811c-45bc-49ff-a3e2-debaedc77a89" containerName="maas-api"
Oct 30 22:30:35.236178 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.236105    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.335211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.331323    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-e90906-openshift-default-dockercfg-kt4hf\""
Oct 30 22:30:35.501223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.452999    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-token\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.501223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.453062    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz447\" (UniqueName: \"kubernetes.io/projected/c73136ea-01b3-4ded-ab76-1c902be19f3a-kube-api-access-fz447\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.501223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.453105    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-credential-socket\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.501223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.453157    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-workload-certs\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.501223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.453245    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-workload-socket\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.501223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.453279    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-data\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.501223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.453315    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73136ea-01b3-4ded-ab76-1c902be19f3a-istiod-ca-cert\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.501223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.453488    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-podinfo\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.501223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.453546    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-envoy\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.627433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.608487    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-podinfo\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.627433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.608592    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-envoy\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.627433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.608624    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-token\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.627433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.608668    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz447\" (UniqueName: \"kubernetes.io/projected/c73136ea-01b3-4ded-ab76-1c902be19f3a-kube-api-access-fz447\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.627433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.608712    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-credential-socket\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.627433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.608742    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-workload-certs\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.627433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.608836    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-workload-socket\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.627433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.608868    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-data\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.627433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.608902    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73136ea-01b3-4ded-ab76-1c902be19f3a-istiod-ca-cert\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.627433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.609838    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73136ea-01b3-4ded-ab76-1c902be19f3a-istiod-ca-cert\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.627433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.626470    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-workload-certs\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.651624 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.651551    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-workload-socket\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.652121 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.652075    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-data\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.669575 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.669531    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-credential-socket\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.707170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.684054    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"]
Oct 30 22:30:35.707493 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.706772    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-envoy\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.755224 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.742614    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-podinfo\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.785161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.768739    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz447\" (UniqueName: \"kubernetes.io/projected/c73136ea-01b3-4ded-ab76-1c902be19f3a-kube-api-access-fz447\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.787524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.787457    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-token\") pod \"e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") " pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:35.887449 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:35.887101    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:36.404164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:36.403260    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:30:36.853230 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:36.841922    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:30:36.967181 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:36.915226    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:30:37.179415 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.178615    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"]
Oct 30 22:30:37.248718 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:30:37.248669    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73136ea_01b3_4ded_ab76_1c902be19f3a.slice/crio-e74d012f96ca7ee695b32716caedc8d96a7d200284ba562b5e80527efee90902 WatchSource:0}: Error finding container e74d012f96ca7ee695b32716caedc8d96a7d200284ba562b5e80527efee90902: Status 404 returned error can't find the container with id e74d012f96ca7ee695b32716caedc8d96a7d200284ba562b5e80527efee90902
Oct 30 22:30:37.257744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.256773    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:30:37.257744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.256865    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:30:37.257744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.256904    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:30:37.651248 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.651215    4553 generic.go:356] "Generic (PLEG): container finished" podID="6a77b4ea-e62c-4f91-9264-d037dbe9210e" containerID="0ec280cf0f4e5a8b58349f718d6b866e3db51dd564e623539534340a8136fc03" exitCode=0
Oct 30 22:30:37.651636 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.651419    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc" event={"ID":"6a77b4ea-e62c-4f91-9264-d037dbe9210e","Type":"ContainerDied","Data":"0ec280cf0f4e5a8b58349f718d6b866e3db51dd564e623539534340a8136fc03"}
Oct 30 22:30:37.651636 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.651467    4553 scope.go:122] "RemoveContainer" containerID="77d73cb21519996a3c577fe7a525a04b5d5cd26be8d2645ce40d781c3370170e"
Oct 30 22:30:37.655385 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.654644    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4" event={"ID":"c73136ea-01b3-4ded-ab76-1c902be19f3a","Type":"ContainerStarted","Data":"f39cb08c1b8b9a973b90a03215655c7a8a7d0ba8f4e91cc85305315418e0f6f9"}
Oct 30 22:30:37.655385 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.654690    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4" event={"ID":"c73136ea-01b3-4ded-ab76-1c902be19f3a","Type":"ContainerStarted","Data":"e74d012f96ca7ee695b32716caedc8d96a7d200284ba562b5e80527efee90902"}
Oct 30 22:30:37.821388 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.820207    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:30:37.888709 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.888668    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:37.890066 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.890044    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istiod-ca-cert\") pod \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") "
Oct 30 22:30:37.890164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.890076    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-token\" (UniqueName: \"kubernetes.io/projected/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-token\") pod \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") "
Oct 30 22:30:37.890164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.890108    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-podinfo\") pod \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") "
Oct 30 22:30:37.890285 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.890177    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-workload-socket\") pod \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") "
Oct 30 22:30:37.890285 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.890197    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-workload-certs\") pod \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") "
Oct 30 22:30:37.890527 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.890230    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-data\") pod \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") "
Oct 30 22:30:37.890644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.890538    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-workload-socket" pod "6a77b4ea-e62c-4f91-9264-d037dbe9210e" (UID: "6a77b4ea-e62c-4f91-9264-d037dbe9210e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:37.890644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.890584    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istiod-ca-cert" pod "6a77b4ea-e62c-4f91-9264-d037dbe9210e" (UID: "6a77b4ea-e62c-4f91-9264-d037dbe9210e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:37.890752 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.890680    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-data" pod "6a77b4ea-e62c-4f91-9264-d037dbe9210e" (UID: "6a77b4ea-e62c-4f91-9264-d037dbe9210e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:37.890752 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.890742    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-credential-socket\") pod \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") "
Oct 30 22:30:37.890850 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.890760    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-workload-certs" pod "6a77b4ea-e62c-4f91-9264-d037dbe9210e" (UID: "6a77b4ea-e62c-4f91-9264-d037dbe9210e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:37.890850 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.890774    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-envoy\") pod \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") "
Oct 30 22:30:37.890960 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.890864    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a77b4ea-e62c-4f91-9264-d037dbe9210e-kube-api-access-mgmfc\" (UniqueName: \"kubernetes.io/projected/6a77b4ea-e62c-4f91-9264-d037dbe9210e-kube-api-access-mgmfc\") pod \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\" (UID: \"6a77b4ea-e62c-4f91-9264-d037dbe9210e\") "
Oct 30 22:30:37.891020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.890975    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-credential-socket" pod "6a77b4ea-e62c-4f91-9264-d037dbe9210e" (UID: "6a77b4ea-e62c-4f91-9264-d037dbe9210e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:37.891430 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.891410    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:37.891518 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.891431    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:37.891518 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.891441    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:37.891518 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.891449    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:37.891518 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.891459    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:37.893103 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.893074    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-token" pod "6a77b4ea-e62c-4f91-9264-d037dbe9210e" (UID: "6a77b4ea-e62c-4f91-9264-d037dbe9210e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:37.893380 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.893362    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-envoy" pod "6a77b4ea-e62c-4f91-9264-d037dbe9210e" (UID: "6a77b4ea-e62c-4f91-9264-d037dbe9210e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:37.893429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.893387    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-podinfo" pod "6a77b4ea-e62c-4f91-9264-d037dbe9210e" (UID: "6a77b4ea-e62c-4f91-9264-d037dbe9210e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:37.894629 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.894605    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a77b4ea-e62c-4f91-9264-d037dbe9210e-kube-api-access-mgmfc" pod "6a77b4ea-e62c-4f91-9264-d037dbe9210e" (UID: "6a77b4ea-e62c-4f91-9264-d037dbe9210e"). InnerVolumeSpecName "kube-api-access-mgmfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:37.966163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.959383    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4" podStartSLOduration=2.959360347 podStartE2EDuration="2.959360347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:37.786417481 +0000 UTC m=+2888.520780513" watchObservedRunningTime="2026-10-30 22:30:37.959360347 +0000 UTC m=+2888.693723379"
Oct 30 22:30:37.996749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.996362    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:37.996749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.996408    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgmfc\" (UniqueName: \"kubernetes.io/projected/6a77b4ea-e62c-4f91-9264-d037dbe9210e-kube-api-access-mgmfc\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:37.996749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.996427    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:37.996749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:37.996440    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a77b4ea-e62c-4f91-9264-d037dbe9210e-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:38.663366 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:38.663333    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"
Oct 30 22:30:38.663887 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:38.663367    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc" event={"ID":"6a77b4ea-e62c-4f91-9264-d037dbe9210e","Type":"ContainerDied","Data":"9aa8cecee2f1f133db9759df7c37750bf71fcca39c88ededd0e93cb89d68ca07"}
Oct 30 22:30:38.663887 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:38.663439    4553 scope.go:122] "RemoveContainer" containerID="0ec280cf0f4e5a8b58349f718d6b866e3db51dd564e623539534340a8136fc03"
Oct 30 22:30:38.843534 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:38.843496    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"]
Oct 30 22:30:38.893326 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:38.893288    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a853fe-openshift-default-6c854c8bb5-6rpbc"]
Oct 30 22:30:38.894355 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:38.894334    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:39.394620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:39.394587    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a77b4ea-e62c-4f91-9264-d037dbe9210e" path="/var/lib/kubelet/pods/6a77b4ea-e62c-4f91-9264-d037dbe9210e/volumes"
Oct 30 22:30:39.670901 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:39.670814    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:39.671986 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:39.671966    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:30:42.673222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.673186    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"]
Oct 30 22:30:42.676512 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.676486    4553 cpu_manager.go:414] "containerMap: removing container" podUID="6a77b4ea-e62c-4f91-9264-d037dbe9210e" containerName="istio-proxy"
Oct 30 22:30:42.679051 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.677657    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a77b4ea-e62c-4f91-9264-d037dbe9210e" containerName="istio-proxy"
Oct 30 22:30:42.679252 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.679237    4553 cpu_manager.go:414] "containerMap: removing container" podUID="6a77b4ea-e62c-4f91-9264-d037dbe9210e" containerName="istio-proxy"
Oct 30 22:30:42.679348 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.679337    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a77b4ea-e62c-4f91-9264-d037dbe9210e" containerName="istio-proxy"
Oct 30 22:30:42.679746 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.679724    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a77b4ea-e62c-4f91-9264-d037dbe9210e" containerName="istio-proxy"
Oct 30 22:30:42.679746 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.679748    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a77b4ea-e62c-4f91-9264-d037dbe9210e" containerName="istio-proxy"
Oct 30 22:30:42.679945 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.679767    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e727db7-7a5b-445d-9b6f-75d186911229" containerName="cleanup"
Oct 30 22:30:42.685756 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.685729    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"
Oct 30 22:30:42.733725 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.733676    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"]
Oct 30 22:30:42.739636 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.739589    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-a0116e\""
Oct 30 22:30:42.739820 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.739732    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-a0116e-dockercfg-flgt4\""
Oct 30 22:30:42.779248 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.779201    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310c1cdc-3c1b-4d84-9f50-9ac5e59e6198-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl\" (UID: \"310c1cdc-3c1b-4d84-9f50-9ac5e59e6198\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"
Oct 30 22:30:42.779456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.779330    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdhch\" (UniqueName: \"kubernetes.io/projected/310c1cdc-3c1b-4d84-9f50-9ac5e59e6198-kube-api-access-cdhch\") pod \"payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl\" (UID: \"310c1cdc-3c1b-4d84-9f50-9ac5e59e6198\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"
Oct 30 22:30:42.880273 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.880221    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdhch\" (UniqueName: \"kubernetes.io/projected/310c1cdc-3c1b-4d84-9f50-9ac5e59e6198-kube-api-access-cdhch\") pod \"payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl\" (UID: \"310c1cdc-3c1b-4d84-9f50-9ac5e59e6198\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"
Oct 30 22:30:42.880509 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.880424    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310c1cdc-3c1b-4d84-9f50-9ac5e59e6198-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl\" (UID: \"310c1cdc-3c1b-4d84-9f50-9ac5e59e6198\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"
Oct 30 22:30:42.881330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.881308    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310c1cdc-3c1b-4d84-9f50-9ac5e59e6198-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl\" (UID: \"310c1cdc-3c1b-4d84-9f50-9ac5e59e6198\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"
Oct 30 22:30:42.946689 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:42.946592    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdhch\" (UniqueName: \"kubernetes.io/projected/310c1cdc-3c1b-4d84-9f50-9ac5e59e6198-kube-api-access-cdhch\") pod \"payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl\" (UID: \"310c1cdc-3c1b-4d84-9f50-9ac5e59e6198\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"
Oct 30 22:30:43.003890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:43.003851    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"
Oct 30 22:30:43.416919 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:43.416890    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"]
Oct 30 22:30:43.419891 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:30:43.419859    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310c1cdc_3c1b_4d84_9f50_9ac5e59e6198.slice/crio-37f3bfbe84e72ccf88bbea798f9825c13a403044a2b28a871260df3778ba7fd1 WatchSource:0}: Error finding container 37f3bfbe84e72ccf88bbea798f9825c13a403044a2b28a871260df3778ba7fd1: Status 404 returned error can't find the container with id 37f3bfbe84e72ccf88bbea798f9825c13a403044a2b28a871260df3778ba7fd1
Oct 30 22:30:43.704290 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:43.704192    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl" event={"ID":"310c1cdc-3c1b-4d84-9f50-9ac5e59e6198","Type":"ContainerStarted","Data":"bd4b766fb4b7a5e0bfc5edb6084de5a3c8b07e3918fc1996638247d8d53b75e0"}
Oct 30 22:30:43.704290 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:43.704259    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"
Oct 30 22:30:43.704290 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:43.704283    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl" event={"ID":"310c1cdc-3c1b-4d84-9f50-9ac5e59e6198","Type":"ContainerStarted","Data":"37f3bfbe84e72ccf88bbea798f9825c13a403044a2b28a871260df3778ba7fd1"}
Oct 30 22:30:43.819031 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:43.818982    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl" podStartSLOduration=1.818967814 podStartE2EDuration="1.818967814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:43.818928057 +0000 UTC m=+2894.553291087" watchObservedRunningTime="2026-10-30 22:30:43.818967814 +0000 UTC m=+2894.553330839"
Oct 30 22:30:45.614241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:45.614211    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"]
Oct 30 22:30:45.723102 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:45.723067    4553 generic.go:356] "Generic (PLEG): container finished" podID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" containerID="53f25c23bcbec20b6c880861c6745417f308881364a7bd824e1d48fffac04ab4" exitCode=0
Oct 30 22:30:45.723425 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:45.723122    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44" event={"ID":"f8af1bfa-6dda-45a6-9435-1ec9b4afd530","Type":"ContainerDied","Data":"53f25c23bcbec20b6c880861c6745417f308881364a7bd824e1d48fffac04ab4"}
Oct 30 22:30:45.724003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:45.723942    4553 scope.go:122] "RemoveContainer" containerID="53f25c23bcbec20b6c880861c6745417f308881364a7bd824e1d48fffac04ab4"
Oct 30 22:30:45.725678 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:45.725654    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"]
Oct 30 22:30:45.725798 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:45.725745    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:30:45.725847 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:45.725817    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:30:45.725896 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:45.725852    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:30:46.740549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:46.740510    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44" event={"ID":"f8af1bfa-6dda-45a6-9435-1ec9b4afd530","Type":"ContainerStarted","Data":"2208d28a03a537b9bd63ada2fce778c6e8d75a9be00482ecb8ed2e4446bed13d"}
Oct 30 22:30:46.796669 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:46.796626    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:30:46.796863 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:46.796828    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 22:30:46.796932 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:46.796881    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44" podUID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 22:30:47.796732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:47.796688    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 22:30:47.797251 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:47.796747    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44" podUID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 22:30:48.797068 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:48.797024    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 22:30:48.797611 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:48.797083    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44" podUID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 22:30:49.797727 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.797690    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 22:30:49.798223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.797749    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44" podUID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 22:30:49.837808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.837773    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"]
Oct 30 22:30:49.842299 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.842275    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.895337 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.895294    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.895554 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.895362    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnzkm\" (UniqueName: \"kubernetes.io/projected/607d78fc-1484-4719-beb9-e6f32376729d-kube-api-access-lnzkm\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.895554 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.895482    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.895554 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.895512    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/607d78fc-1484-4719-beb9-e6f32376729d-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.895676 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.895608    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.895713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.895674    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.895784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.895771    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/607d78fc-1484-4719-beb9-e6f32376729d-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.895821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.895798    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.895925 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.895910    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/607d78fc-1484-4719-beb9-e6f32376729d-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.940371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.940336    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"]
Oct 30 22:30:49.997782 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.997747    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.997999 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.997798    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnzkm\" (UniqueName: \"kubernetes.io/projected/607d78fc-1484-4719-beb9-e6f32376729d-kube-api-access-lnzkm\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.997999 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.997887    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.997999 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.997912    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/607d78fc-1484-4719-beb9-e6f32376729d-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.998233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.998044    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.998233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.998195    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.998349 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.998304    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/607d78fc-1484-4719-beb9-e6f32376729d-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.998349 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.998341    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.998461 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.998376    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-workload-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.998461 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.998449    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-credential-socket\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.998575 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.998458    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/607d78fc-1484-4719-beb9-e6f32376729d-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.998775 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.998752    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-istio-data\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.998775 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.998761    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-workload-certs\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:49.999062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:49.999042    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/607d78fc-1484-4719-beb9-e6f32376729d-istiod-ca-cert\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:50.000860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.000830    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-istio-envoy\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:50.001452 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.001433    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/607d78fc-1484-4719-beb9-e6f32376729d-istio-podinfo\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:50.055815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.055719    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/607d78fc-1484-4719-beb9-e6f32376729d-istio-token\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:50.055979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.055808    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnzkm\" (UniqueName: \"kubernetes.io/projected/607d78fc-1484-4719-beb9-e6f32376729d-kube-api-access-lnzkm\") pod \"e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") " pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:50.158151 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.158098    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:50.629197 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.629051    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf" podUID="025c2f88-9c30-4930-93ca-4a3305dbc53e" containerName="payload-processing" containerID="cri-o://85361e16c4f8f998f91b1a23ba60468e76171cbdf24708e613b918e2fd578154" gracePeriod=25
Oct 30 22:30:50.639188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.639105    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"]
Oct 30 22:30:50.730458 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.729197    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm" podUID="23f5c585-dd28-4678-8318-22ebc5a69e77" containerName="payload-pre-processing" containerID="cri-o://996501539dfc144b7da2566b0a0003e1d10f737e5402836e592b841b86e5259b" gracePeriod=25
Oct 30 22:30:50.741664 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.741627    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:30:50.741800 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.741708    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:30:50.741800 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.741740    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:30:50.772527 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.772473    4553 generic.go:356] "Generic (PLEG): container finished" podID="025c2f88-9c30-4930-93ca-4a3305dbc53e" containerID="85361e16c4f8f998f91b1a23ba60468e76171cbdf24708e613b918e2fd578154" exitCode=0
Oct 30 22:30:50.772835 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.772546    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf" event={"ID":"025c2f88-9c30-4930-93ca-4a3305dbc53e","Type":"ContainerDied","Data":"85361e16c4f8f998f91b1a23ba60468e76171cbdf24708e613b918e2fd578154"}
Oct 30 22:30:50.775017 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.774970    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9" event={"ID":"607d78fc-1484-4719-beb9-e6f32376729d","Type":"ContainerStarted","Data":"1e26d893ae18b5e260b0f833c3a2fe5e924b7e54fc865fdfc52e01508cba0eac"}
Oct 30 22:30:50.797630 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.797593    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 22:30:50.797830 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.797671    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44" podUID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 22:30:50.967851 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.967811    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"]
Oct 30 22:30:50.968020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.967997    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c" podUID="73f4ca98-5d19-4c61-a5bd-6421cd5198d9" containerName="maas-api" containerID="cri-o://394fc1eab9c846d098b36d135af03b2a5b5a610ff1b6a07e70c91b96804e924d" gracePeriod=30
Oct 30 22:30:50.982376 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:50.982342    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"
Oct 30 22:30:51.053690 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.051089    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025c2f88-9c30-4930-93ca-4a3305dbc53e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025c2f88-9c30-4930-93ca-4a3305dbc53e-plugins-config-volume\") pod \"025c2f88-9c30-4930-93ca-4a3305dbc53e\" (UID: \"025c2f88-9c30-4930-93ca-4a3305dbc53e\") "
Oct 30 22:30:51.053690 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.051412    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025c2f88-9c30-4930-93ca-4a3305dbc53e-kube-api-access-64mq4\" (UniqueName: \"kubernetes.io/projected/025c2f88-9c30-4930-93ca-4a3305dbc53e-kube-api-access-64mq4\") pod \"025c2f88-9c30-4930-93ca-4a3305dbc53e\" (UID: \"025c2f88-9c30-4930-93ca-4a3305dbc53e\") "
Oct 30 22:30:51.053690 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.052987    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025c2f88-9c30-4930-93ca-4a3305dbc53e-plugins-config-volume" pod "025c2f88-9c30-4930-93ca-4a3305dbc53e" (UID: "025c2f88-9c30-4930-93ca-4a3305dbc53e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:51.058976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.058925    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025c2f88-9c30-4930-93ca-4a3305dbc53e-kube-api-access-64mq4" pod "025c2f88-9c30-4930-93ca-4a3305dbc53e" (UID: "025c2f88-9c30-4930-93ca-4a3305dbc53e"). InnerVolumeSpecName "kube-api-access-64mq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:51.153788 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.153755    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64mq4\" (UniqueName: \"kubernetes.io/projected/025c2f88-9c30-4930-93ca-4a3305dbc53e-kube-api-access-64mq4\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.153788 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.153792    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025c2f88-9c30-4930-93ca-4a3305dbc53e-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.185917 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.185885    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"
Oct 30 22:30:51.260026 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.259997    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f5c585-dd28-4678-8318-22ebc5a69e77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f5c585-dd28-4678-8318-22ebc5a69e77-plugins-config-volume\") pod \"23f5c585-dd28-4678-8318-22ebc5a69e77\" (UID: \"23f5c585-dd28-4678-8318-22ebc5a69e77\") "
Oct 30 22:30:51.264903 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.264878    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f5c585-dd28-4678-8318-22ebc5a69e77-kube-api-access-m4zgk\" (UniqueName: \"kubernetes.io/projected/23f5c585-dd28-4678-8318-22ebc5a69e77-kube-api-access-m4zgk\") pod \"23f5c585-dd28-4678-8318-22ebc5a69e77\" (UID: \"23f5c585-dd28-4678-8318-22ebc5a69e77\") "
Oct 30 22:30:51.265814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.265686    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f5c585-dd28-4678-8318-22ebc5a69e77-plugins-config-volume" pod "23f5c585-dd28-4678-8318-22ebc5a69e77" (UID: "23f5c585-dd28-4678-8318-22ebc5a69e77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:51.266937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.266916    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23f5c585-dd28-4678-8318-22ebc5a69e77-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.271543 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.270743    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f5c585-dd28-4678-8318-22ebc5a69e77-kube-api-access-m4zgk" pod "23f5c585-dd28-4678-8318-22ebc5a69e77" (UID: "23f5c585-dd28-4678-8318-22ebc5a69e77"). InnerVolumeSpecName "kube-api-access-m4zgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:51.370346 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.369004    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4zgk\" (UniqueName: \"kubernetes.io/projected/23f5c585-dd28-4678-8318-22ebc5a69e77-kube-api-access-m4zgk\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.392200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.383114    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:30:51.469687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.469593    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-maas-api-tls\") pod \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") "
Oct 30 22:30:51.469687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.469673    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-openshift-service-ca\") pod \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") "
Oct 30 22:30:51.469937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.469705    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-metrics-certs\") pod \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") "
Oct 30 22:30:51.469937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.469879    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-kube-api-access-xpw6q\" (UniqueName: \"kubernetes.io/projected/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-kube-api-access-xpw6q\") pod \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\" (UID: \"73f4ca98-5d19-4c61-a5bd-6421cd5198d9\") "
Oct 30 22:30:51.471230 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.471198    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-openshift-service-ca" pod "73f4ca98-5d19-4c61-a5bd-6421cd5198d9" (UID: "73f4ca98-5d19-4c61-a5bd-6421cd5198d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:51.474262 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.474229    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-kube-api-access-xpw6q" pod "73f4ca98-5d19-4c61-a5bd-6421cd5198d9" (UID: "73f4ca98-5d19-4c61-a5bd-6421cd5198d9"). InnerVolumeSpecName "kube-api-access-xpw6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:51.475071 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.475042    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-maas-api-tls" pod "73f4ca98-5d19-4c61-a5bd-6421cd5198d9" (UID: "73f4ca98-5d19-4c61-a5bd-6421cd5198d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:51.476101 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.476070    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-metrics-certs" pod "73f4ca98-5d19-4c61-a5bd-6421cd5198d9" (UID: "73f4ca98-5d19-4c61-a5bd-6421cd5198d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:51.572411 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.572367    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.572411 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.572416    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.572627 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.572433    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.572627 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.572448    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpw6q\" (UniqueName: \"kubernetes.io/projected/73f4ca98-5d19-4c61-a5bd-6421cd5198d9-kube-api-access-xpw6q\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:30:51.784532 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.784444    4553 generic.go:356] "Generic (PLEG): container finished" podID="73f4ca98-5d19-4c61-a5bd-6421cd5198d9" containerID="394fc1eab9c846d098b36d135af03b2a5b5a610ff1b6a07e70c91b96804e924d" exitCode=0
Oct 30 22:30:51.784532 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.784512    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"
Oct 30 22:30:51.784782 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.784539    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c" event={"ID":"73f4ca98-5d19-4c61-a5bd-6421cd5198d9","Type":"ContainerDied","Data":"394fc1eab9c846d098b36d135af03b2a5b5a610ff1b6a07e70c91b96804e924d"}
Oct 30 22:30:51.784782 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.784600    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c" event={"ID":"73f4ca98-5d19-4c61-a5bd-6421cd5198d9","Type":"ContainerDied","Data":"281bca97efc3074c9792368d2f826d2bea2b8cc333d5aad3964ca1d1e9f88eb5"}
Oct 30 22:30:51.784782 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.784663    4553 scope.go:122] "RemoveContainer" containerID="394fc1eab9c846d098b36d135af03b2a5b5a610ff1b6a07e70c91b96804e924d"
Oct 30 22:30:51.786226 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.786206    4553 generic.go:356] "Generic (PLEG): container finished" podID="23f5c585-dd28-4678-8318-22ebc5a69e77" containerID="996501539dfc144b7da2566b0a0003e1d10f737e5402836e592b841b86e5259b" exitCode=0
Oct 30 22:30:51.786302 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.786270    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"
Oct 30 22:30:51.786302 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.786287    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm" event={"ID":"23f5c585-dd28-4678-8318-22ebc5a69e77","Type":"ContainerDied","Data":"996501539dfc144b7da2566b0a0003e1d10f737e5402836e592b841b86e5259b"}
Oct 30 22:30:51.786485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.786325    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm" event={"ID":"23f5c585-dd28-4678-8318-22ebc5a69e77","Type":"ContainerDied","Data":"95b9c03c218e21cbf9eb1cd22bbf44439a91cd09741eba206dc6876fe3985e40"}
Oct 30 22:30:51.794162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.792404    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"
Oct 30 22:30:51.794162 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.792297    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf" event={"ID":"025c2f88-9c30-4930-93ca-4a3305dbc53e","Type":"ContainerDied","Data":"678e12af6ece389c09bce1d30ed47ef54fcccfa8f333153e4840a94c7738120b"}
Oct 30 22:30:51.794701 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.794676    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9" event={"ID":"607d78fc-1484-4719-beb9-e6f32376729d","Type":"ContainerStarted","Data":"c9f3c1ebcf3d550e7f9781d147f952c03b5849e7775976447d1f9d7bcaeb0bdf"}
Oct 30 22:30:51.797165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.797119    4553 scope.go:122] "RemoveContainer" containerID="394fc1eab9c846d098b36d135af03b2a5b5a610ff1b6a07e70c91b96804e924d"
Oct 30 22:30:51.797378 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.797359    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 22:30:51.797466 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.797403    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44" podUID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 22:30:51.797466 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:30:51.797427    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"394fc1eab9c846d098b36d135af03b2a5b5a610ff1b6a07e70c91b96804e924d\": container with ID starting with 394fc1eab9c846d098b36d135af03b2a5b5a610ff1b6a07e70c91b96804e924d not found: ID does not exist" containerID="394fc1eab9c846d098b36d135af03b2a5b5a610ff1b6a07e70c91b96804e924d"
Oct 30 22:30:51.797466 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.797453    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"394fc1eab9c846d098b36d135af03b2a5b5a610ff1b6a07e70c91b96804e924d"} err="failed to get container status \"394fc1eab9c846d098b36d135af03b2a5b5a610ff1b6a07e70c91b96804e924d\": rpc error: code = NotFound desc = could not find container \"394fc1eab9c846d098b36d135af03b2a5b5a610ff1b6a07e70c91b96804e924d\": container with ID starting with 394fc1eab9c846d098b36d135af03b2a5b5a610ff1b6a07e70c91b96804e924d not found: ID does not exist"
Oct 30 22:30:51.797582 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.797476    4553 scope.go:122] "RemoveContainer" containerID="996501539dfc144b7da2566b0a0003e1d10f737e5402836e592b841b86e5259b"
Oct 30 22:30:51.841180 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.841113    4553 scope.go:122] "RemoveContainer" containerID="996501539dfc144b7da2566b0a0003e1d10f737e5402836e592b841b86e5259b"
Oct 30 22:30:51.841617 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:30:51.841588    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"996501539dfc144b7da2566b0a0003e1d10f737e5402836e592b841b86e5259b\": container with ID starting with 996501539dfc144b7da2566b0a0003e1d10f737e5402836e592b841b86e5259b not found: ID does not exist" containerID="996501539dfc144b7da2566b0a0003e1d10f737e5402836e592b841b86e5259b"
Oct 30 22:30:51.841664 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.841630    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"996501539dfc144b7da2566b0a0003e1d10f737e5402836e592b841b86e5259b"} err="failed to get container status \"996501539dfc144b7da2566b0a0003e1d10f737e5402836e592b841b86e5259b\": rpc error: code = NotFound desc = could not find container \"996501539dfc144b7da2566b0a0003e1d10f737e5402836e592b841b86e5259b\": container with ID starting with 996501539dfc144b7da2566b0a0003e1d10f737e5402836e592b841b86e5259b not found: ID does not exist"
Oct 30 22:30:51.841664 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.841656    4553 scope.go:122] "RemoveContainer" containerID="85361e16c4f8f998f91b1a23ba60468e76171cbdf24708e613b918e2fd578154"
Oct 30 22:30:51.952511 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.952471    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"]
Oct 30 22:30:51.955506 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:51.955476    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b563fd-9877fb8f-4n8nf"]
Oct 30 22:30:52.064811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.064771    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"]
Oct 30 22:30:52.068305 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.068278    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b563fd-669b78595-tpnvm"]
Oct 30 22:30:52.230201 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.229305    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:52.230201 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.229382    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"]
Oct 30 22:30:52.253161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.246258    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:30:52.253161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.246333    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9" podUID="607d78fc-1484-4719-beb9-e6f32376729d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 22:30:52.262571 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.262277    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-086575-c466fbd4-wd76c"]
Oct 30 22:30:52.348189 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.347967    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9" podStartSLOduration=3.347945378 podStartE2EDuration="3.347945378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:52.33244062 +0000 UTC m=+2903.066803652" watchObservedRunningTime="2026-10-30 22:30:52.347945378 +0000 UTC m=+2903.082308410"
Oct 30 22:30:52.440946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.440899    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"]
Oct 30 22:30:52.441900 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.441876    4553 cpu_manager.go:414] "containerMap: removing container" podUID="73f4ca98-5d19-4c61-a5bd-6421cd5198d9" containerName="maas-api"
Oct 30 22:30:52.441900 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.441902    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f4ca98-5d19-4c61-a5bd-6421cd5198d9" containerName="maas-api"
Oct 30 22:30:52.442101 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.441926    4553 cpu_manager.go:414] "containerMap: removing container" podUID="23f5c585-dd28-4678-8318-22ebc5a69e77" containerName="payload-pre-processing"
Oct 30 22:30:52.442101 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.441934    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f5c585-dd28-4678-8318-22ebc5a69e77" containerName="payload-pre-processing"
Oct 30 22:30:52.442101 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.441956    4553 cpu_manager.go:414] "containerMap: removing container" podUID="025c2f88-9c30-4930-93ca-4a3305dbc53e" containerName="payload-processing"
Oct 30 22:30:52.442101 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.441964    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025c2f88-9c30-4930-93ca-4a3305dbc53e" containerName="payload-processing"
Oct 30 22:30:52.442390 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.442159    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f4ca98-5d19-4c61-a5bd-6421cd5198d9" containerName="maas-api"
Oct 30 22:30:52.442390 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.442173    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f5c585-dd28-4678-8318-22ebc5a69e77" containerName="payload-pre-processing"
Oct 30 22:30:52.442390 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.442186    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="025c2f88-9c30-4930-93ca-4a3305dbc53e" containerName="payload-processing"
Oct 30 22:30:52.447747 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.447721    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"
Oct 30 22:30:52.490464 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.490433    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"]
Oct 30 22:30:52.501125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.501088    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-e90906-dockercfg-t6khb\""
Oct 30 22:30:52.501775 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.501755    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-e90906\""
Oct 30 22:30:52.584126 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.584084    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"]
Oct 30 22:30:52.605571 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.605487    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"
Oct 30 22:30:52.607619 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.607584    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3d82a4-6522-40f9-a000-e92b7b95aa76-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q\" (UID: \"8a3d82a4-6522-40f9-a000-e92b7b95aa76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"
Oct 30 22:30:52.607981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.607962    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7s6t\" (UniqueName: \"kubernetes.io/projected/8a3d82a4-6522-40f9-a000-e92b7b95aa76-kube-api-access-z7s6t\") pod \"payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q\" (UID: \"8a3d82a4-6522-40f9-a000-e92b7b95aa76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"
Oct 30 22:30:52.654174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.651681    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"]
Oct 30 22:30:52.756453 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.756406    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3d82a4-6522-40f9-a000-e92b7b95aa76-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q\" (UID: \"8a3d82a4-6522-40f9-a000-e92b7b95aa76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"
Oct 30 22:30:52.756668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.756516    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7s6t\" (UniqueName: \"kubernetes.io/projected/8a3d82a4-6522-40f9-a000-e92b7b95aa76-kube-api-access-z7s6t\") pod \"payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q\" (UID: \"8a3d82a4-6522-40f9-a000-e92b7b95aa76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"
Oct 30 22:30:52.756668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.756554    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrk6\" (UniqueName: \"kubernetes.io/projected/417967ed-ef2e-449f-95fe-fadf4e6926e5-kube-api-access-tjrk6\") pod \"payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t\" (UID: \"417967ed-ef2e-449f-95fe-fadf4e6926e5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"
Oct 30 22:30:52.756668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.756582    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/417967ed-ef2e-449f-95fe-fadf4e6926e5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t\" (UID: \"417967ed-ef2e-449f-95fe-fadf4e6926e5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"
Oct 30 22:30:52.757416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.757390    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3d82a4-6522-40f9-a000-e92b7b95aa76-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q\" (UID: \"8a3d82a4-6522-40f9-a000-e92b7b95aa76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"
Oct 30 22:30:52.802858 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.802819    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:30:52.803354 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.803328    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:30:52.804352 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.804331    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:30:52.834245 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.834207    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7s6t\" (UniqueName: \"kubernetes.io/projected/8a3d82a4-6522-40f9-a000-e92b7b95aa76-kube-api-access-z7s6t\") pod \"payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q\" (UID: \"8a3d82a4-6522-40f9-a000-e92b7b95aa76\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"
Oct 30 22:30:52.858113 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.858021    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrk6\" (UniqueName: \"kubernetes.io/projected/417967ed-ef2e-449f-95fe-fadf4e6926e5-kube-api-access-tjrk6\") pod \"payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t\" (UID: \"417967ed-ef2e-449f-95fe-fadf4e6926e5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"
Oct 30 22:30:52.858616 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.858593    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/417967ed-ef2e-449f-95fe-fadf4e6926e5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t\" (UID: \"417967ed-ef2e-449f-95fe-fadf4e6926e5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"
Oct 30 22:30:52.859350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.859326    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/417967ed-ef2e-449f-95fe-fadf4e6926e5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t\" (UID: \"417967ed-ef2e-449f-95fe-fadf4e6926e5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"
Oct 30 22:30:52.938701 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.938666    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrk6\" (UniqueName: \"kubernetes.io/projected/417967ed-ef2e-449f-95fe-fadf4e6926e5-kube-api-access-tjrk6\") pod \"payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t\" (UID: \"417967ed-ef2e-449f-95fe-fadf4e6926e5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"
Oct 30 22:30:52.971607 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:52.971570    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"
Oct 30 22:30:53.072082 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:53.071822    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"
Oct 30 22:30:53.159494 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:53.159416    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:30:53.159494 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:53.159471    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9" podUID="607d78fc-1484-4719-beb9-e6f32376729d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 22:30:53.376700 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:53.376574    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"]
Oct 30 22:30:53.377779 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:30:53.377748    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417967ed_ef2e_449f_95fe_fadf4e6926e5.slice/crio-89f2d5be915a25c91de29a42cd90b4e65f568086f2c8012f285e71c4a54cf95c WatchSource:0}: Error finding container 89f2d5be915a25c91de29a42cd90b4e65f568086f2c8012f285e71c4a54cf95c: Status 404 returned error can't find the container with id 89f2d5be915a25c91de29a42cd90b4e65f568086f2c8012f285e71c4a54cf95c
Oct 30 22:30:53.403021 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:53.402989    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025c2f88-9c30-4930-93ca-4a3305dbc53e" path="/var/lib/kubelet/pods/025c2f88-9c30-4930-93ca-4a3305dbc53e/volumes"
Oct 30 22:30:53.403674 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:53.403656    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f5c585-dd28-4678-8318-22ebc5a69e77" path="/var/lib/kubelet/pods/23f5c585-dd28-4678-8318-22ebc5a69e77/volumes"
Oct 30 22:30:53.404231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:53.404215    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f4ca98-5d19-4c61-a5bd-6421cd5198d9" path="/var/lib/kubelet/pods/73f4ca98-5d19-4c61-a5bd-6421cd5198d9/volumes"
Oct 30 22:30:53.534586 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:53.534002    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"]
Oct 30 22:30:53.541751 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:30:53.541700    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3d82a4_6522_40f9_a000_e92b7b95aa76.slice/crio-84af5e95940b170023ce99dfee00293389d50ce162828b64a2697fd0652b5ef1 WatchSource:0}: Error finding container 84af5e95940b170023ce99dfee00293389d50ce162828b64a2697fd0652b5ef1: Status 404 returned error can't find the container with id 84af5e95940b170023ce99dfee00293389d50ce162828b64a2697fd0652b5ef1
Oct 30 22:30:53.840736 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:53.840697    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t" event={"ID":"417967ed-ef2e-449f-95fe-fadf4e6926e5","Type":"ContainerStarted","Data":"fc50e97c7b83bf97fe00fee61b59d0ca938a37ea0655cd5823fa6a8584db2500"}
Oct 30 22:30:53.840948 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:53.840765    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"
Oct 30 22:30:53.840948 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:53.840788    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t" event={"ID":"417967ed-ef2e-449f-95fe-fadf4e6926e5","Type":"ContainerStarted","Data":"89f2d5be915a25c91de29a42cd90b4e65f568086f2c8012f285e71c4a54cf95c"}
Oct 30 22:30:53.843833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:53.843796    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q" event={"ID":"8a3d82a4-6522-40f9-a000-e92b7b95aa76","Type":"ContainerStarted","Data":"de16a9c2f1639636b15130b2e6af76bf9b6ef4edb98b82e0efbcc5ce9c2de3d0"}
Oct 30 22:30:53.844056 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:53.843862    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q" event={"ID":"8a3d82a4-6522-40f9-a000-e92b7b95aa76","Type":"ContainerStarted","Data":"84af5e95940b170023ce99dfee00293389d50ce162828b64a2697fd0652b5ef1"}
Oct 30 22:30:53.844457 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:53.844438    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"
Oct 30 22:30:53.959660 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:53.959593    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t" podStartSLOduration=1.959577483 podStartE2EDuration="1.959577483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:53.956089044 +0000 UTC m=+2904.690452076" watchObservedRunningTime="2026-10-30 22:30:53.959577483 +0000 UTC m=+2904.693940543"
Oct 30 22:30:54.082153 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:54.081471    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q" podStartSLOduration=2.081453221 podStartE2EDuration="2.081453221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:54.079727848 +0000 UTC m=+2904.814090879" watchObservedRunningTime="2026-10-30 22:30:54.081453221 +0000 UTC m=+2904.815816252"
Oct 30 22:30:54.159373 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:54.159288    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:30:54.159373 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:54.159345    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9" podUID="607d78fc-1484-4719-beb9-e6f32376729d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 22:30:54.733851 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:54.733675    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"
Oct 30 22:30:55.159280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:55.159239    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:30:55.159686 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:55.159300    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9" podUID="607d78fc-1484-4719-beb9-e6f32376729d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 22:30:56.159543 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:56.159509    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:30:56.159974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:56.159569    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9" podUID="607d78fc-1484-4719-beb9-e6f32376729d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 22:30:57.158862 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:57.158822    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 22:30:57.159065 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:57.158878    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9" podUID="607d78fc-1484-4719-beb9-e6f32376729d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 22:30:58.206667 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:30:58.206623    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:30:58.611565 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:58.611534    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:58.611985 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:58.611960    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:58.612985 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:58.612963    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:30:58.718420 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:30:58.718384    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"]
Oct 30 22:31:04.859455 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:04.859421    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"
Oct 30 22:31:04.876378 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:04.876349    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"
Oct 30 22:31:13.740651 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:13.740538    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf" podUID="2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7" containerName="main" containerID="cri-o://b5cc6b0e6c8e532474a355cbf6307b25ac468f44b45a26e2aa34fcdf8219885a" gracePeriod=45
Oct 30 22:31:13.996632 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:13.996556    4553 generic.go:356] "Generic (PLEG): container finished" podID="2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7" containerID="b5cc6b0e6c8e532474a355cbf6307b25ac468f44b45a26e2aa34fcdf8219885a" exitCode=0
Oct 30 22:31:13.996807 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:13.996624    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf" event={"ID":"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7","Type":"ContainerDied","Data":"b5cc6b0e6c8e532474a355cbf6307b25ac468f44b45a26e2aa34fcdf8219885a"}
Oct 30 22:31:14.026721 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.026696    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:31:14.207608 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.207574    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-model-cache\") pod \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") "
Oct 30 22:31:14.207608 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.207612    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-dshm\") pod \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") "
Oct 30 22:31:14.207871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.207753    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-tls-certs\") pod \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") "
Oct 30 22:31:14.207871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.207791    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-kube-api-access-l78tn\" (UniqueName: \"kubernetes.io/projected/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-kube-api-access-l78tn\") pod \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") "
Oct 30 22:31:14.207871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.207847    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-tmp-dir\") pod \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") "
Oct 30 22:31:14.208045 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.207899    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-model-cache" pod "2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7" (UID: "2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:14.208045 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.207944    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-home\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-home\") pod \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\" (UID: \"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7\") "
Oct 30 22:31:14.208272 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.208236    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-tmp-dir" pod "2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7" (UID: "2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:14.208434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.208382    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-home" pod "2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7" (UID: "2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:14.208898 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.208873    4553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-model-cache\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:14.209011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.208903    4553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-tmp-dir\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:14.209011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.208919    4553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-home\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:14.210347 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.210323    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-dshm" pod "2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7" (UID: "2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:14.210463 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.210402    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-tls-certs" pod "2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7" (UID: "2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:14.210522 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.210465    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-kube-api-access-l78tn" pod "2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7" (UID: "2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7"). InnerVolumeSpecName "kube-api-access-l78tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:14.310524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.310438    4553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-dshm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:14.310524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.310469    4553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-tls-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:14.310524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:14.310480    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l78tn\" (UniqueName: \"kubernetes.io/projected/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7-kube-api-access-l78tn\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:15.004634 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:15.004607    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"
Oct 30 22:31:15.005106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:15.004649    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf" event={"ID":"2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7","Type":"ContainerDied","Data":"9355e20e88d53e5c50d5a44bb25f2e2fcde10af68f91041b34a6796db413084b"}
Oct 30 22:31:15.005106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:15.004700    4553 scope.go:122] "RemoveContainer" containerID="b5cc6b0e6c8e532474a355cbf6307b25ac468f44b45a26e2aa34fcdf8219885a"
Oct 30 22:31:15.172212 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:15.172168    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"]
Oct 30 22:31:15.222672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:15.222633    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-086575/praxis-route-086575-kserve-945fdf7dc-2pdmf"]
Oct 30 22:31:15.395007 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:15.394973    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7" path="/var/lib/kubelet/pods/2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7/volumes"
Oct 30 22:31:28.205299 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:31:28.205258    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:31:34.684524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.684484    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"]
Oct 30 22:31:34.685191 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.685175    4553 cpu_manager.go:414] "containerMap: removing container" podUID="2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7" containerName="main"
Oct 30 22:31:34.685249 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.685192    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7" containerName="main"
Oct 30 22:31:34.685320 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.685311    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc9ec8d-22b0-4f4b-aa46-91b76653e0d7" containerName="main"
Oct 30 22:31:34.690206 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.690189    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:34.745688 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.745656    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"]
Oct 30 22:31:34.788595 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.788556    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:34.788795 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.788600    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9rq8\" (UniqueName: \"kubernetes.io/projected/71914545-402c-416e-b1a8-99f17216e7fe-kube-api-access-r9rq8\") pod \"maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:34.788795 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.788689    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71914545-402c-416e-b1a8-99f17216e7fe-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:34.788903 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.788879    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:34.792338 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.792318    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-e90906-serving-cert\""
Oct 30 22:31:34.889828 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.889789    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:34.890025 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.889921    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:34.890025 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.889947    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9rq8\" (UniqueName: \"kubernetes.io/projected/71914545-402c-416e-b1a8-99f17216e7fe-kube-api-access-r9rq8\") pod \"maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:34.890025 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:31:34.889964    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-serving-cert: secret "maas-api-e2e-shared-a-w0-e90906-serving-cert" not found
Oct 30 22:31:34.890025 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.889975    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71914545-402c-416e-b1a8-99f17216e7fe-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:34.890224 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:31:34.890054    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-maas-api-tls podName:71914545-402c-416e-b1a8-99f17216e7fe nodeName:}" failed. No retries permitted until 2026-10-30 22:31:35.390031486 +0000 UTC m=+2946.124394519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-maas-api-tls") pod "maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7" (UID: "71914545-402c-416e-b1a8-99f17216e7fe") : secret "maas-api-e2e-shared-a-w0-e90906-serving-cert" not found
Oct 30 22:31:34.890802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.890727    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71914545-402c-416e-b1a8-99f17216e7fe-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:34.892941 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.892912    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:34.949573 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:34.949495    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9rq8\" (UniqueName: \"kubernetes.io/projected/71914545-402c-416e-b1a8-99f17216e7fe-kube-api-access-r9rq8\") pod \"maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:35.400033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:35.400000    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:35.402705 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:35.402682    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:35.604729 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:35.604691    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:35.983026 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:35.982996    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"]
Oct 30 22:31:35.983741 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:31:35.983718    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71914545_402c_416e_b1a8_99f17216e7fe.slice/crio-cb0b4b046062ac00ff031ddade9609f1dd7b10590a57401bdce2ef761304d980 WatchSource:0}: Error finding container cb0b4b046062ac00ff031ddade9609f1dd7b10590a57401bdce2ef761304d980: Status 404 returned error can't find the container with id cb0b4b046062ac00ff031ddade9609f1dd7b10590a57401bdce2ef761304d980
Oct 30 22:31:36.145460 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:36.145422    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7" event={"ID":"71914545-402c-416e-b1a8-99f17216e7fe","Type":"ContainerStarted","Data":"cb0b4b046062ac00ff031ddade9609f1dd7b10590a57401bdce2ef761304d980"}
Oct 30 22:31:37.153527 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:37.153494    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7" event={"ID":"71914545-402c-416e-b1a8-99f17216e7fe","Type":"ContainerStarted","Data":"ef47eb06da3859d316d355bf6f783a4c91fcc73459e24aab3b218ebb5b495250"}
Oct 30 22:31:37.153954 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:37.153564    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:37.275769 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:37.275705    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7" podStartSLOduration=2.462245731 podStartE2EDuration="3.275687579s" totalImagesPullingTime="813.441848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:37.275169732 +0000 UTC m=+2948.009532763" watchObservedRunningTime="2026-10-30 22:31:37.275687579 +0000 UTC m=+2948.010050643"
Oct 30 22:31:43.166528 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:43.166499    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:31:52.081908 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:52.081866    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"]
Oct 30 22:31:52.191573 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:52.191538    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"]
Oct 30 22:31:54.637760 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:54.637711    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"]
Oct 30 22:31:54.750090 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:54.750054    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"]
Oct 30 22:31:54.750598 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:54.750544    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t" podUID="a72b7e51-d030-4462-824b-a6ddbf89ea26" containerName="maas-api" containerID="cri-o://5174c4e2924c75dd7b5392521c540f5191010ab957f97889523af6a4aa4970f3" gracePeriod=30
Oct 30 22:31:54.754000 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:54.753965    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"]
Oct 30 22:31:55.066833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.066797    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:31:55.185175 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.185050    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-maas-api-tls\") pod \"a72b7e51-d030-4462-824b-a6ddbf89ea26\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") "
Oct 30 22:31:55.185175 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.185098    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a72b7e51-d030-4462-824b-a6ddbf89ea26-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a72b7e51-d030-4462-824b-a6ddbf89ea26-openshift-service-ca\") pod \"a72b7e51-d030-4462-824b-a6ddbf89ea26\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") "
Oct 30 22:31:55.185175 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.185153    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-metrics-certs\") pod \"a72b7e51-d030-4462-824b-a6ddbf89ea26\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") "
Oct 30 22:31:55.185498 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.185206    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72b7e51-d030-4462-824b-a6ddbf89ea26-kube-api-access-dfnbx\" (UniqueName: \"kubernetes.io/projected/a72b7e51-d030-4462-824b-a6ddbf89ea26-kube-api-access-dfnbx\") pod \"a72b7e51-d030-4462-824b-a6ddbf89ea26\" (UID: \"a72b7e51-d030-4462-824b-a6ddbf89ea26\") "
Oct 30 22:31:55.185989 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.185959    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72b7e51-d030-4462-824b-a6ddbf89ea26-openshift-service-ca" pod "a72b7e51-d030-4462-824b-a6ddbf89ea26" (UID: "a72b7e51-d030-4462-824b-a6ddbf89ea26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:55.188048 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.188017    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-maas-api-tls" pod "a72b7e51-d030-4462-824b-a6ddbf89ea26" (UID: "a72b7e51-d030-4462-824b-a6ddbf89ea26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:55.188317 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.188294    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72b7e51-d030-4462-824b-a6ddbf89ea26-kube-api-access-dfnbx" pod "a72b7e51-d030-4462-824b-a6ddbf89ea26" (UID: "a72b7e51-d030-4462-824b-a6ddbf89ea26"). InnerVolumeSpecName "kube-api-access-dfnbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:55.188462 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.188439    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-metrics-certs" pod "a72b7e51-d030-4462-824b-a6ddbf89ea26" (UID: "a72b7e51-d030-4462-824b-a6ddbf89ea26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:55.270786 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.270749    4553 generic.go:356] "Generic (PLEG): container finished" podID="a72b7e51-d030-4462-824b-a6ddbf89ea26" containerID="5174c4e2924c75dd7b5392521c540f5191010ab957f97889523af6a4aa4970f3" exitCode=0
Oct 30 22:31:55.270969 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.270797    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"
Oct 30 22:31:55.270969 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.270822    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t" event={"ID":"a72b7e51-d030-4462-824b-a6ddbf89ea26","Type":"ContainerDied","Data":"5174c4e2924c75dd7b5392521c540f5191010ab957f97889523af6a4aa4970f3"}
Oct 30 22:31:55.270969 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.270873    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t" event={"ID":"a72b7e51-d030-4462-824b-a6ddbf89ea26","Type":"ContainerDied","Data":"2dba3dd33bc4c3a7091432b82a37e09fee7e4c7c4daad81a8d5df908c42928ec"}
Oct 30 22:31:55.270969 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.270896    4553 scope.go:122] "RemoveContainer" containerID="5174c4e2924c75dd7b5392521c540f5191010ab957f97889523af6a4aa4970f3"
Oct 30 22:31:55.282251 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.282231    4553 scope.go:122] "RemoveContainer" containerID="5174c4e2924c75dd7b5392521c540f5191010ab957f97889523af6a4aa4970f3"
Oct 30 22:31:55.282534 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:31:55.282513    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5174c4e2924c75dd7b5392521c540f5191010ab957f97889523af6a4aa4970f3\": container with ID starting with 5174c4e2924c75dd7b5392521c540f5191010ab957f97889523af6a4aa4970f3 not found: ID does not exist" containerID="5174c4e2924c75dd7b5392521c540f5191010ab957f97889523af6a4aa4970f3"
Oct 30 22:31:55.282592 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.282544    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5174c4e2924c75dd7b5392521c540f5191010ab957f97889523af6a4aa4970f3"} err="failed to get container status \"5174c4e2924c75dd7b5392521c540f5191010ab957f97889523af6a4aa4970f3\": rpc error: code = NotFound desc = could not find container \"5174c4e2924c75dd7b5392521c540f5191010ab957f97889523af6a4aa4970f3\": container with ID starting with 5174c4e2924c75dd7b5392521c540f5191010ab957f97889523af6a4aa4970f3 not found: ID does not exist"
Oct 30 22:31:55.287453 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.287433    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:55.287533 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.287455    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a72b7e51-d030-4462-824b-a6ddbf89ea26-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:55.287533 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.287466    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a72b7e51-d030-4462-824b-a6ddbf89ea26-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:55.287533 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.287475    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfnbx\" (UniqueName: \"kubernetes.io/projected/a72b7e51-d030-4462-824b-a6ddbf89ea26-kube-api-access-dfnbx\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:55.437495 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.437404    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"]
Oct 30 22:31:55.443415 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:55.443384    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t"]
Oct 30 22:31:57.082743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.082678    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt" podUID="df13fb2a-c588-45a7-a5bc-b21a5b8aec6f" containerName="payload-processing" containerID="cri-o://eaebea7c7144475446153503a9300931ce9189b657d5ab9ac72c648045eef33e" gracePeriod=25
Oct 30 22:31:57.192536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.192486    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz" podUID="6106ed06-a521-4806-8090-78724db50f2c" containerName="payload-pre-processing" containerID="cri-o://cb8654ed9090d37b201e5124002382b6b76a280b73304a2557e4f0a2ab9f8640" gracePeriod=25
Oct 30 22:31:57.289431 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.289401    4553 generic.go:356] "Generic (PLEG): container finished" podID="df13fb2a-c588-45a7-a5bc-b21a5b8aec6f" containerID="eaebea7c7144475446153503a9300931ce9189b657d5ab9ac72c648045eef33e" exitCode=0
Oct 30 22:31:57.289597 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.289444    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt" event={"ID":"df13fb2a-c588-45a7-a5bc-b21a5b8aec6f","Type":"ContainerDied","Data":"eaebea7c7144475446153503a9300931ce9189b657d5ab9ac72c648045eef33e"}
Oct 30 22:31:57.371435 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.371406    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"
Oct 30 22:31:57.396741 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.396703    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a72b7e51-d030-4462-824b-a6ddbf89ea26" path="/var/lib/kubelet/pods/a72b7e51-d030-4462-824b-a6ddbf89ea26/volumes"
Oct 30 22:31:57.492428 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.492402    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"
Oct 30 22:31:57.521589 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.521560    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df13fb2a-c588-45a7-a5bc-b21a5b8aec6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df13fb2a-c588-45a7-a5bc-b21a5b8aec6f-plugins-config-volume\") pod \"df13fb2a-c588-45a7-a5bc-b21a5b8aec6f\" (UID: \"df13fb2a-c588-45a7-a5bc-b21a5b8aec6f\") "
Oct 30 22:31:57.521776 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.521597    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df13fb2a-c588-45a7-a5bc-b21a5b8aec6f-kube-api-access-q7h2q\" (UniqueName: \"kubernetes.io/projected/df13fb2a-c588-45a7-a5bc-b21a5b8aec6f-kube-api-access-q7h2q\") pod \"df13fb2a-c588-45a7-a5bc-b21a5b8aec6f\" (UID: \"df13fb2a-c588-45a7-a5bc-b21a5b8aec6f\") "
Oct 30 22:31:57.522052 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.522023    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df13fb2a-c588-45a7-a5bc-b21a5b8aec6f-plugins-config-volume" pod "df13fb2a-c588-45a7-a5bc-b21a5b8aec6f" (UID: "df13fb2a-c588-45a7-a5bc-b21a5b8aec6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:57.523972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.523949    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df13fb2a-c588-45a7-a5bc-b21a5b8aec6f-kube-api-access-q7h2q" pod "df13fb2a-c588-45a7-a5bc-b21a5b8aec6f" (UID: "df13fb2a-c588-45a7-a5bc-b21a5b8aec6f"). InnerVolumeSpecName "kube-api-access-q7h2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:57.622497 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.622418    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6106ed06-a521-4806-8090-78724db50f2c-kube-api-access-ncprb\" (UniqueName: \"kubernetes.io/projected/6106ed06-a521-4806-8090-78724db50f2c-kube-api-access-ncprb\") pod \"6106ed06-a521-4806-8090-78724db50f2c\" (UID: \"6106ed06-a521-4806-8090-78724db50f2c\") "
Oct 30 22:31:57.622497 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.622471    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6106ed06-a521-4806-8090-78724db50f2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6106ed06-a521-4806-8090-78724db50f2c-plugins-config-volume\") pod \"6106ed06-a521-4806-8090-78724db50f2c\" (UID: \"6106ed06-a521-4806-8090-78724db50f2c\") "
Oct 30 22:31:57.622892 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.622857    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6106ed06-a521-4806-8090-78724db50f2c-plugins-config-volume" pod "6106ed06-a521-4806-8090-78724db50f2c" (UID: "6106ed06-a521-4806-8090-78724db50f2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:57.623378 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.623300    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6106ed06-a521-4806-8090-78724db50f2c-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:57.623435 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.623385    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df13fb2a-c588-45a7-a5bc-b21a5b8aec6f-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:57.623435 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.623400    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7h2q\" (UniqueName: \"kubernetes.io/projected/df13fb2a-c588-45a7-a5bc-b21a5b8aec6f-kube-api-access-q7h2q\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:57.624982 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.624948    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6106ed06-a521-4806-8090-78724db50f2c-kube-api-access-ncprb" pod "6106ed06-a521-4806-8090-78724db50f2c" (UID: "6106ed06-a521-4806-8090-78724db50f2c"). InnerVolumeSpecName "kube-api-access-ncprb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:57.724535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.724496    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncprb\" (UniqueName: \"kubernetes.io/projected/6106ed06-a521-4806-8090-78724db50f2c-kube-api-access-ncprb\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:31:57.998725 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:57.998628    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-42e8a1-5cdcbb6c88-xwh5t" podUID="a72b7e51-d030-4462-824b-a6ddbf89ea26" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.178:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:31:58.202374 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:31:58.202333    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:31:58.296696 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:58.296599    4553 generic.go:356] "Generic (PLEG): container finished" podID="6106ed06-a521-4806-8090-78724db50f2c" containerID="cb8654ed9090d37b201e5124002382b6b76a280b73304a2557e4f0a2ab9f8640" exitCode=0
Oct 30 22:31:58.296696 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:58.296650    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"
Oct 30 22:31:58.296696 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:58.296684    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz" event={"ID":"6106ed06-a521-4806-8090-78724db50f2c","Type":"ContainerDied","Data":"cb8654ed9090d37b201e5124002382b6b76a280b73304a2557e4f0a2ab9f8640"}
Oct 30 22:31:58.296971 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:58.296744    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz" event={"ID":"6106ed06-a521-4806-8090-78724db50f2c","Type":"ContainerDied","Data":"d0ba39666bf404f12ed6dc0a48f1e8383c73fcab7ed620498955c30af55873f6"}
Oct 30 22:31:58.296971 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:58.296771    4553 scope.go:122] "RemoveContainer" containerID="cb8654ed9090d37b201e5124002382b6b76a280b73304a2557e4f0a2ab9f8640"
Oct 30 22:31:58.298470 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:58.298435    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt" event={"ID":"df13fb2a-c588-45a7-a5bc-b21a5b8aec6f","Type":"ContainerDied","Data":"7f460e4429e953269bc6c56e3efd62d9690d9fe2d8a02ab02a155d88cca5be8b"}
Oct 30 22:31:58.298603 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:58.298583    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"
Oct 30 22:31:58.308981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:58.308962    4553 scope.go:122] "RemoveContainer" containerID="cb8654ed9090d37b201e5124002382b6b76a280b73304a2557e4f0a2ab9f8640"
Oct 30 22:31:58.309327 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:31:58.309307    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8654ed9090d37b201e5124002382b6b76a280b73304a2557e4f0a2ab9f8640\": container with ID starting with cb8654ed9090d37b201e5124002382b6b76a280b73304a2557e4f0a2ab9f8640 not found: ID does not exist" containerID="cb8654ed9090d37b201e5124002382b6b76a280b73304a2557e4f0a2ab9f8640"
Oct 30 22:31:58.309420 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:58.309335    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8654ed9090d37b201e5124002382b6b76a280b73304a2557e4f0a2ab9f8640"} err="failed to get container status \"cb8654ed9090d37b201e5124002382b6b76a280b73304a2557e4f0a2ab9f8640\": rpc error: code = NotFound desc = could not find container \"cb8654ed9090d37b201e5124002382b6b76a280b73304a2557e4f0a2ab9f8640\": container with ID starting with cb8654ed9090d37b201e5124002382b6b76a280b73304a2557e4f0a2ab9f8640 not found: ID does not exist"
Oct 30 22:31:58.309420 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:58.309353    4553 scope.go:122] "RemoveContainer" containerID="eaebea7c7144475446153503a9300931ce9189b657d5ab9ac72c648045eef33e"
Oct 30 22:31:58.463435 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:58.463402    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"]
Oct 30 22:31:58.543165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:58.539754    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-42e8a1-b87d5f9dc-jq2qz"]
Oct 30 22:31:58.635004 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:58.634970    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"]
Oct 30 22:31:58.643797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:58.643770    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-42e8a1-67dbdc88b9-f55lt"]
Oct 30 22:31:59.396475 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.396445    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6106ed06-a521-4806-8090-78724db50f2c" path="/var/lib/kubelet/pods/6106ed06-a521-4806-8090-78724db50f2c/volumes"
Oct 30 22:31:59.396891 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.396878    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df13fb2a-c588-45a7-a5bc-b21a5b8aec6f" path="/var/lib/kubelet/pods/df13fb2a-c588-45a7-a5bc-b21a5b8aec6f/volumes"
Oct 30 22:31:59.638239 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.638196    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5" podUID="b1433d5b-9380-414f-a649-82fb38e66302" containerName="payload-processing" containerID="cri-o://a137b2f302f979ab62787e9ca9e267198b24f15e1694c8f1cb20e6f175bb5e1d" gracePeriod=25
Oct 30 22:31:59.738516 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.738484    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"]
Oct 30 22:31:59.739173 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.739157    4553 cpu_manager.go:414] "containerMap: removing container" podUID="df13fb2a-c588-45a7-a5bc-b21a5b8aec6f" containerName="payload-processing"
Oct 30 22:31:59.739219 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.739174    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df13fb2a-c588-45a7-a5bc-b21a5b8aec6f" containerName="payload-processing"
Oct 30 22:31:59.739219 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.739187    4553 cpu_manager.go:414] "containerMap: removing container" podUID="6106ed06-a521-4806-8090-78724db50f2c" containerName="payload-pre-processing"
Oct 30 22:31:59.739219 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.739192    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6106ed06-a521-4806-8090-78724db50f2c" containerName="payload-pre-processing"
Oct 30 22:31:59.739330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.739221    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a72b7e51-d030-4462-824b-a6ddbf89ea26" containerName="maas-api"
Oct 30 22:31:59.739330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.739227    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a72b7e51-d030-4462-824b-a6ddbf89ea26" containerName="maas-api"
Oct 30 22:31:59.739330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.739319    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="df13fb2a-c588-45a7-a5bc-b21a5b8aec6f" containerName="payload-processing"
Oct 30 22:31:59.739330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.739329    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="6106ed06-a521-4806-8090-78724db50f2c" containerName="payload-pre-processing"
Oct 30 22:31:59.739447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.739341    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="a72b7e51-d030-4462-824b-a6ddbf89ea26" containerName="maas-api"
Oct 30 22:31:59.742724 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.742702    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:31:59.755027 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.754990    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8" podUID="ab37b2d1-4c30-4d6f-971f-96195f6a4d27" containerName="payload-pre-processing" containerID="cri-o://eb6689edba14badde0285be3c457005673e667c9dc24bf28c41adce1936671fb" gracePeriod=25
Oct 30 22:31:59.796475 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.796403    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-b4deef-openshift-default-dockercfg-n8wsr\""
Oct 30 22:31:59.911597 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.911508    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-workload-socket\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:31:59.911786 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.911631    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-envoy\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:31:59.911786 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.911666    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-podinfo\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:31:59.911786 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.911733    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-credential-socket\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:31:59.911786 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.911780    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-workload-certs\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:31:59.912005 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.911840    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fbwf\" (UniqueName: \"kubernetes.io/projected/c6e1ba27-d171-44ac-b625-2d2e34fb1481-kube-api-access-4fbwf\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:31:59.912005 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.911887    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istiod-ca-cert\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:31:59.912005 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.911933    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-data\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:31:59.912005 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.911962    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-token\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:31:59.917746 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:31:59.917618    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"]
Oct 30 22:32:00.049167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.033027    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-credential-socket\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.049167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.033186    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-workload-certs\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.049167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.033261    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fbwf\" (UniqueName: \"kubernetes.io/projected/c6e1ba27-d171-44ac-b625-2d2e34fb1481-kube-api-access-4fbwf\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.049167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.033307    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istiod-ca-cert\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.049167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.033352    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-data\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.049167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.033385    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-token\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.049167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.033450    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-workload-socket\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.049167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.033568    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-envoy\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.049167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.033595    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-podinfo\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.058421 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.056866    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-podinfo\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.058421 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.057629    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-data\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.058421 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.057906    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istiod-ca-cert\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.058421 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.058186    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-credential-socket\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.058421 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.058254    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-workload-socket\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.058785 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.058428    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-workload-certs\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.060773 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.060744    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-envoy\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.140164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.140024    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fbwf\" (UniqueName: \"kubernetes.io/projected/c6e1ba27-d171-44ac-b625-2d2e34fb1481-kube-api-access-4fbwf\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.145005 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.144198    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-token\") pod \"e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") " pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.163336 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.163257    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"
Oct 30 22:32:00.244494 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.244462    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"
Oct 30 22:32:00.341101 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.341063    4553 generic.go:356] "Generic (PLEG): container finished" podID="b1433d5b-9380-414f-a649-82fb38e66302" containerID="a137b2f302f979ab62787e9ca9e267198b24f15e1694c8f1cb20e6f175bb5e1d" exitCode=0
Oct 30 22:32:00.341323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.341106    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"
Oct 30 22:32:00.341323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.341175    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5" event={"ID":"b1433d5b-9380-414f-a649-82fb38e66302","Type":"ContainerDied","Data":"a137b2f302f979ab62787e9ca9e267198b24f15e1694c8f1cb20e6f175bb5e1d"}
Oct 30 22:32:00.341323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.341232    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5" event={"ID":"b1433d5b-9380-414f-a649-82fb38e66302","Type":"ContainerDied","Data":"b218a1c6e2fae3953b1eb74d476aedd3d7eaed42a51ad03ec1d399e6125991b3"}
Oct 30 22:32:00.341323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.341261    4553 scope.go:122] "RemoveContainer" containerID="a137b2f302f979ab62787e9ca9e267198b24f15e1694c8f1cb20e6f175bb5e1d"
Oct 30 22:32:00.342230 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.342208    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab37b2d1-4c30-4d6f-971f-96195f6a4d27-kube-api-access-qkc5k\" (UniqueName: \"kubernetes.io/projected/ab37b2d1-4c30-4d6f-971f-96195f6a4d27-kube-api-access-qkc5k\") pod \"ab37b2d1-4c30-4d6f-971f-96195f6a4d27\" (UID: \"ab37b2d1-4c30-4d6f-971f-96195f6a4d27\") "
Oct 30 22:32:00.342408 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.342288    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1433d5b-9380-414f-a649-82fb38e66302-kube-api-access-s8x9b\" (UniqueName: \"kubernetes.io/projected/b1433d5b-9380-414f-a649-82fb38e66302-kube-api-access-s8x9b\") pod \"b1433d5b-9380-414f-a649-82fb38e66302\" (UID: \"b1433d5b-9380-414f-a649-82fb38e66302\") "
Oct 30 22:32:00.342408 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.342361    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab37b2d1-4c30-4d6f-971f-96195f6a4d27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab37b2d1-4c30-4d6f-971f-96195f6a4d27-plugins-config-volume\") pod \"ab37b2d1-4c30-4d6f-971f-96195f6a4d27\" (UID: \"ab37b2d1-4c30-4d6f-971f-96195f6a4d27\") "
Oct 30 22:32:00.342614 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.342447    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1433d5b-9380-414f-a649-82fb38e66302-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1433d5b-9380-414f-a649-82fb38e66302-plugins-config-volume\") pod \"b1433d5b-9380-414f-a649-82fb38e66302\" (UID: \"b1433d5b-9380-414f-a649-82fb38e66302\") "
Oct 30 22:32:00.343223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.343193    4553 generic.go:356] "Generic (PLEG): container finished" podID="ab37b2d1-4c30-4d6f-971f-96195f6a4d27" containerID="eb6689edba14badde0285be3c457005673e667c9dc24bf28c41adce1936671fb" exitCode=0
Oct 30 22:32:00.343402 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.343093    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab37b2d1-4c30-4d6f-971f-96195f6a4d27-plugins-config-volume" pod "ab37b2d1-4c30-4d6f-971f-96195f6a4d27" (UID: "ab37b2d1-4c30-4d6f-971f-96195f6a4d27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:00.343402 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.343263    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8" event={"ID":"ab37b2d1-4c30-4d6f-971f-96195f6a4d27","Type":"ContainerDied","Data":"eb6689edba14badde0285be3c457005673e667c9dc24bf28c41adce1936671fb"}
Oct 30 22:32:00.343402 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.343297    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1433d5b-9380-414f-a649-82fb38e66302-plugins-config-volume" pod "b1433d5b-9380-414f-a649-82fb38e66302" (UID: "b1433d5b-9380-414f-a649-82fb38e66302"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:00.343402 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.343310    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8" event={"ID":"ab37b2d1-4c30-4d6f-971f-96195f6a4d27","Type":"ContainerDied","Data":"934a2c2848d5f2a0b7c453ddb800de46c5c505782589153d3f68456ce6209fd3"}
Oct 30 22:32:00.343402 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.343328    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"
Oct 30 22:32:00.345920 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.345896    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1433d5b-9380-414f-a649-82fb38e66302-kube-api-access-s8x9b" pod "b1433d5b-9380-414f-a649-82fb38e66302" (UID: "b1433d5b-9380-414f-a649-82fb38e66302"). InnerVolumeSpecName "kube-api-access-s8x9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:00.346025 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.345984    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab37b2d1-4c30-4d6f-971f-96195f6a4d27-kube-api-access-qkc5k" pod "ab37b2d1-4c30-4d6f-971f-96195f6a4d27" (UID: "ab37b2d1-4c30-4d6f-971f-96195f6a4d27"). InnerVolumeSpecName "kube-api-access-qkc5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:00.356107 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.356020    4553 scope.go:122] "RemoveContainer" containerID="a137b2f302f979ab62787e9ca9e267198b24f15e1694c8f1cb20e6f175bb5e1d"
Oct 30 22:32:00.357000 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:32:00.356452    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a137b2f302f979ab62787e9ca9e267198b24f15e1694c8f1cb20e6f175bb5e1d\": container with ID starting with a137b2f302f979ab62787e9ca9e267198b24f15e1694c8f1cb20e6f175bb5e1d not found: ID does not exist" containerID="a137b2f302f979ab62787e9ca9e267198b24f15e1694c8f1cb20e6f175bb5e1d"
Oct 30 22:32:00.357265 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.357020    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a137b2f302f979ab62787e9ca9e267198b24f15e1694c8f1cb20e6f175bb5e1d"} err="failed to get container status \"a137b2f302f979ab62787e9ca9e267198b24f15e1694c8f1cb20e6f175bb5e1d\": rpc error: code = NotFound desc = could not find container \"a137b2f302f979ab62787e9ca9e267198b24f15e1694c8f1cb20e6f175bb5e1d\": container with ID starting with a137b2f302f979ab62787e9ca9e267198b24f15e1694c8f1cb20e6f175bb5e1d not found: ID does not exist"
Oct 30 22:32:00.357265 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.357050    4553 scope.go:122] "RemoveContainer" containerID="eb6689edba14badde0285be3c457005673e667c9dc24bf28c41adce1936671fb"
Oct 30 22:32:00.357561 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.357545    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:00.382542 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.382516    4553 scope.go:122] "RemoveContainer" containerID="eb6689edba14badde0285be3c457005673e667c9dc24bf28c41adce1936671fb"
Oct 30 22:32:00.383005 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:32:00.382958    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb6689edba14badde0285be3c457005673e667c9dc24bf28c41adce1936671fb\": container with ID starting with eb6689edba14badde0285be3c457005673e667c9dc24bf28c41adce1936671fb not found: ID does not exist" containerID="eb6689edba14badde0285be3c457005673e667c9dc24bf28c41adce1936671fb"
Oct 30 22:32:00.383071 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.383036    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb6689edba14badde0285be3c457005673e667c9dc24bf28c41adce1936671fb"} err="failed to get container status \"eb6689edba14badde0285be3c457005673e667c9dc24bf28c41adce1936671fb\": rpc error: code = NotFound desc = could not find container \"eb6689edba14badde0285be3c457005673e667c9dc24bf28c41adce1936671fb\": container with ID starting with eb6689edba14badde0285be3c457005673e667c9dc24bf28c41adce1936671fb not found: ID does not exist"
Oct 30 22:32:00.444436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.444339    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkc5k\" (UniqueName: \"kubernetes.io/projected/ab37b2d1-4c30-4d6f-971f-96195f6a4d27-kube-api-access-qkc5k\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:00.444436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.444382    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8x9b\" (UniqueName: \"kubernetes.io/projected/b1433d5b-9380-414f-a649-82fb38e66302-kube-api-access-s8x9b\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:00.444436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.444397    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab37b2d1-4c30-4d6f-971f-96195f6a4d27-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:00.444436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.444411    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1433d5b-9380-414f-a649-82fb38e66302-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:00.790577 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.790494    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"]
Oct 30 22:32:00.794678 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.794647    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-37a390-59bfbfcf5-lh2w5"]
Oct 30 22:32:00.838988 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.838951    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"]
Oct 30 22:32:00.844678 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:32:00.844651    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e1ba27_d171_44ac_b625_2d2e34fb1481.slice/crio-b1236fcc15dcdf264f81d974661e4131fbfb566f63f837e3e45a876a882dadea WatchSource:0}: Error finding container b1236fcc15dcdf264f81d974661e4131fbfb566f63f837e3e45a876a882dadea: Status 404 returned error can't find the container with id b1236fcc15dcdf264f81d974661e4131fbfb566f63f837e3e45a876a882dadea
Oct 30 22:32:00.846805 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.846775    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:32:00.846896 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.846848    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:32:00.846960 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.846905    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:32:00.905799 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.905767    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"]
Oct 30 22:32:00.940194 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:00.937413    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-37a390-795bc4bfd7-x79x8"]
Oct 30 22:32:01.356796 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:01.356760    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp" event={"ID":"c6e1ba27-d171-44ac-b625-2d2e34fb1481","Type":"ContainerStarted","Data":"a1f8da1485dd857622bce01acd471fdf203b396d3a42b55d3ab0a9bfefb2056a"}
Oct 30 22:32:01.357020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:01.356819    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp" event={"ID":"c6e1ba27-d171-44ac-b625-2d2e34fb1481","Type":"ContainerStarted","Data":"b1236fcc15dcdf264f81d974661e4131fbfb566f63f837e3e45a876a882dadea"}
Oct 30 22:32:01.358188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:01.358165    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:01.360115 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:01.360083    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 30 22:32:01.360285 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:01.360176    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp" podUID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 30 22:32:01.393150 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:01.393100    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab37b2d1-4c30-4d6f-971f-96195f6a4d27" path="/var/lib/kubelet/pods/ab37b2d1-4c30-4d6f-971f-96195f6a4d27/volumes"
Oct 30 22:32:01.393605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:01.393588    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1433d5b-9380-414f-a649-82fb38e66302" path="/var/lib/kubelet/pods/b1433d5b-9380-414f-a649-82fb38e66302/volumes"
Oct 30 22:32:01.482029 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:01.481974    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp" podStartSLOduration=2.4819587 podStartE2EDuration="2.4819587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:01.47980916 +0000 UTC m=+2972.214172189" watchObservedRunningTime="2026-10-30 22:32:01.4819587 +0000 UTC m=+2972.216321802"
Oct 30 22:32:02.364767 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:02.364741    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:03.369921 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:03.369885    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:03.370983 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:03.370963    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:32:09.434281 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:09.434242    4553 generic.go:356] "Generic (PLEG): container finished" podID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerID="f39cb08c1b8b9a973b90a03215655c7a8a7d0ba8f4e91cc85305315418e0f6f9" exitCode=0
Oct 30 22:32:09.434764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:09.434325    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4" event={"ID":"c73136ea-01b3-4ded-ab76-1c902be19f3a","Type":"ContainerDied","Data":"f39cb08c1b8b9a973b90a03215655c7a8a7d0ba8f4e91cc85305315418e0f6f9"}
Oct 30 22:32:09.434867 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:09.434852    4553 scope.go:122] "RemoveContainer" containerID="f39cb08c1b8b9a973b90a03215655c7a8a7d0ba8f4e91cc85305315418e0f6f9"
Oct 30 22:32:09.436737 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:09.436708    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:32:09.436812 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:09.436782    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:32:09.436852 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:09.436816    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:32:09.671593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:09.671563    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:32:10.443653 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:10.443619    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4" event={"ID":"c73136ea-01b3-4ded-ab76-1c902be19f3a","Type":"ContainerStarted","Data":"d0be27e66cb690257f3c0a03762b7c42b36de707ac285ad43d66364def4f4db0"}
Oct 30 22:32:10.888443 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:10.888411    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:32:10.890444 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:10.890049    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:32:10.890444 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:10.890166    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4" podUID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 22:32:11.888182 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:11.888117    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:32:11.888618 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:11.888224    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4" podUID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 22:32:12.889085 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:12.889048    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:32:12.889556 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:12.889109    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4" podUID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 22:32:13.889060 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:13.889002    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:32:13.889393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:13.889077    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4" podUID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 22:32:14.888962 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:14.888924    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:32:14.889203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:14.888980    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4" podUID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 22:32:15.891580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:15.891533    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 22:32:15.892234 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:15.892199    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4" podUID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 22:32:16.000541 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.000502    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"]
Oct 30 22:32:16.001184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.001168    4553 cpu_manager.go:414] "containerMap: removing container" podUID="ab37b2d1-4c30-4d6f-971f-96195f6a4d27" containerName="payload-pre-processing"
Oct 30 22:32:16.001184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.001184    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab37b2d1-4c30-4d6f-971f-96195f6a4d27" containerName="payload-pre-processing"
Oct 30 22:32:16.001300 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.001198    4553 cpu_manager.go:414] "containerMap: removing container" podUID="b1433d5b-9380-414f-a649-82fb38e66302" containerName="payload-processing"
Oct 30 22:32:16.001300 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.001204    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1433d5b-9380-414f-a649-82fb38e66302" containerName="payload-processing"
Oct 30 22:32:16.001300 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.001296    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab37b2d1-4c30-4d6f-971f-96195f6a4d27" containerName="payload-pre-processing"
Oct 30 22:32:16.001396 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.001303    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1433d5b-9380-414f-a649-82fb38e66302" containerName="payload-processing"
Oct 30 22:32:16.005024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.005001    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"
Oct 30 22:32:16.060393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.060357    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"]
Oct 30 22:32:16.080112 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.080079    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fed9e2-b815-4444-b5dc-db25fb1a42c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h\" (UID: \"a2fed9e2-b815-4444-b5dc-db25fb1a42c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"
Oct 30 22:32:16.080324 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.080303    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btpxg\" (UniqueName: \"kubernetes.io/projected/a2fed9e2-b815-4444-b5dc-db25fb1a42c5-kube-api-access-btpxg\") pod \"payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h\" (UID: \"a2fed9e2-b815-4444-b5dc-db25fb1a42c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"
Oct 30 22:32:16.156223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.141228    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-b4deef\""
Oct 30 22:32:16.156476 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.142293    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-b4deef-dockercfg-v6l88\""
Oct 30 22:32:16.156599 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.150108    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"]
Oct 30 22:32:16.162437 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.162390    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"
Oct 30 22:32:16.183213 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.183179    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fed9e2-b815-4444-b5dc-db25fb1a42c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h\" (UID: \"a2fed9e2-b815-4444-b5dc-db25fb1a42c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"
Oct 30 22:32:16.183635 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.183613    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btpxg\" (UniqueName: \"kubernetes.io/projected/a2fed9e2-b815-4444-b5dc-db25fb1a42c5-kube-api-access-btpxg\") pod \"payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h\" (UID: \"a2fed9e2-b815-4444-b5dc-db25fb1a42c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"
Oct 30 22:32:16.185189 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.185034    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fed9e2-b815-4444-b5dc-db25fb1a42c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h\" (UID: \"a2fed9e2-b815-4444-b5dc-db25fb1a42c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"
Oct 30 22:32:16.214544 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.214504    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"]
Oct 30 22:32:16.246468 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.246428    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btpxg\" (UniqueName: \"kubernetes.io/projected/a2fed9e2-b815-4444-b5dc-db25fb1a42c5-kube-api-access-btpxg\") pod \"payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h\" (UID: \"a2fed9e2-b815-4444-b5dc-db25fb1a42c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"
Oct 30 22:32:16.285079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.285043    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4eb8f4c-df16-4758-a3dd-cab73631f335-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl\" (UID: \"b4eb8f4c-df16-4758-a3dd-cab73631f335\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"
Oct 30 22:32:16.285297 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.285204    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck2mg\" (UniqueName: \"kubernetes.io/projected/b4eb8f4c-df16-4758-a3dd-cab73631f335-kube-api-access-ck2mg\") pod \"payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl\" (UID: \"b4eb8f4c-df16-4758-a3dd-cab73631f335\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"
Oct 30 22:32:16.335896 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.335862    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"
Oct 30 22:32:16.386903 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.386861    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4eb8f4c-df16-4758-a3dd-cab73631f335-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl\" (UID: \"b4eb8f4c-df16-4758-a3dd-cab73631f335\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"
Oct 30 22:32:16.387080 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.387006    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck2mg\" (UniqueName: \"kubernetes.io/projected/b4eb8f4c-df16-4758-a3dd-cab73631f335-kube-api-access-ck2mg\") pod \"payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl\" (UID: \"b4eb8f4c-df16-4758-a3dd-cab73631f335\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"
Oct 30 22:32:16.387671 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.387653    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4eb8f4c-df16-4758-a3dd-cab73631f335-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl\" (UID: \"b4eb8f4c-df16-4758-a3dd-cab73631f335\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"
Oct 30 22:32:16.445837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.445810    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck2mg\" (UniqueName: \"kubernetes.io/projected/b4eb8f4c-df16-4758-a3dd-cab73631f335-kube-api-access-ck2mg\") pod \"payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl\" (UID: \"b4eb8f4c-df16-4758-a3dd-cab73631f335\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"
Oct 30 22:32:16.498574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.498543    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"
Oct 30 22:32:16.723479 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.723450    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"]
Oct 30 22:32:16.724157 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:32:16.724117    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fed9e2_b815_4444_b5dc_db25fb1a42c5.slice/crio-05d5c9fb0bb18d816f6d20e3dd22fa7cc6901c6ccde494493b5a427412ee1ecc WatchSource:0}: Error finding container 05d5c9fb0bb18d816f6d20e3dd22fa7cc6901c6ccde494493b5a427412ee1ecc: Status 404 returned error can't find the container with id 05d5c9fb0bb18d816f6d20e3dd22fa7cc6901c6ccde494493b5a427412ee1ecc
Oct 30 22:32:16.874759 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.874727    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"]
Oct 30 22:32:16.875232 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:32:16.875207    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4eb8f4c_df16_4758_a3dd_cab73631f335.slice/crio-6292d9043d926c11fc12c2825107504d20938892c8b502ca2d9ce6251b02437d WatchSource:0}: Error finding container 6292d9043d926c11fc12c2825107504d20938892c8b502ca2d9ce6251b02437d: Status 404 returned error can't find the container with id 6292d9043d926c11fc12c2825107504d20938892c8b502ca2d9ce6251b02437d
Oct 30 22:32:16.893668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.893645    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:32:16.894219 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.894193    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:32:16.895036 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:16.895019    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:32:17.493447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:17.493400    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl" event={"ID":"b4eb8f4c-df16-4758-a3dd-cab73631f335","Type":"ContainerStarted","Data":"eaa124c0435ad9f74018acfc1bb37e71294e9233a1c61d2b7308b2fc05597d60"}
Oct 30 22:32:17.493658 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:17.493476    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"
Oct 30 22:32:17.493658 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:17.493501    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl" event={"ID":"b4eb8f4c-df16-4758-a3dd-cab73631f335","Type":"ContainerStarted","Data":"6292d9043d926c11fc12c2825107504d20938892c8b502ca2d9ce6251b02437d"}
Oct 30 22:32:17.495188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:17.495125    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h" event={"ID":"a2fed9e2-b815-4444-b5dc-db25fb1a42c5","Type":"ContainerStarted","Data":"c72ee1a43f69fe736721c968bb299c188411bb9a5a891e9ba0bb5c1a6972647d"}
Oct 30 22:32:17.495325 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:17.495210    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"
Oct 30 22:32:17.495325 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:17.495227    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h" event={"ID":"a2fed9e2-b815-4444-b5dc-db25fb1a42c5","Type":"ContainerStarted","Data":"05d5c9fb0bb18d816f6d20e3dd22fa7cc6901c6ccde494493b5a427412ee1ecc"}
Oct 30 22:32:17.611515 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:17.611450    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl" podStartSLOduration=1.611433 podStartE2EDuration="1.611433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:17.610286246 +0000 UTC m=+2988.344649274" watchObservedRunningTime="2026-10-30 22:32:17.611433 +0000 UTC m=+2988.345796031"
Oct 30 22:32:17.732844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:17.732778    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h" podStartSLOduration=2.732760591 podStartE2EDuration="2.732760591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:17.729113766 +0000 UTC m=+2988.463476798" watchObservedRunningTime="2026-10-30 22:32:17.732760591 +0000 UTC m=+2988.467123625"
Oct 30 22:32:19.181331 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:19.181287    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"]
Oct 30 22:32:24.182404 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.182337    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5" podUID="f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b" containerName="payload-pre-processing" containerID="cri-o://8849ff46b633c1f9948870fe9185cb1456095f3fbdc73144d8a2d721f56208a5" gracePeriod=25
Oct 30 22:32:24.453054 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.452988    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"
Oct 30 22:32:24.552786 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.552756    4553 generic.go:356] "Generic (PLEG): container finished" podID="f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b" containerID="8849ff46b633c1f9948870fe9185cb1456095f3fbdc73144d8a2d721f56208a5" exitCode=0
Oct 30 22:32:24.552994 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.552803    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"
Oct 30 22:32:24.552994 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.552849    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5" event={"ID":"f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b","Type":"ContainerDied","Data":"8849ff46b633c1f9948870fe9185cb1456095f3fbdc73144d8a2d721f56208a5"}
Oct 30 22:32:24.552994 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.552918    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5" event={"ID":"f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b","Type":"ContainerDied","Data":"2623b607564ea4e9a3fcdc5392458a248242e099bbda13f492e1e97c2e1c5a7a"}
Oct 30 22:32:24.552994 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.552950    4553 scope.go:122] "RemoveContainer" containerID="8849ff46b633c1f9948870fe9185cb1456095f3fbdc73144d8a2d721f56208a5"
Oct 30 22:32:24.566471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.566450    4553 scope.go:122] "RemoveContainer" containerID="8849ff46b633c1f9948870fe9185cb1456095f3fbdc73144d8a2d721f56208a5"
Oct 30 22:32:24.566801 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:32:24.566777    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8849ff46b633c1f9948870fe9185cb1456095f3fbdc73144d8a2d721f56208a5\": container with ID starting with 8849ff46b633c1f9948870fe9185cb1456095f3fbdc73144d8a2d721f56208a5 not found: ID does not exist" containerID="8849ff46b633c1f9948870fe9185cb1456095f3fbdc73144d8a2d721f56208a5"
Oct 30 22:32:24.566879 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.566812    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8849ff46b633c1f9948870fe9185cb1456095f3fbdc73144d8a2d721f56208a5"} err="failed to get container status \"8849ff46b633c1f9948870fe9185cb1456095f3fbdc73144d8a2d721f56208a5\": rpc error: code = NotFound desc = could not find container \"8849ff46b633c1f9948870fe9185cb1456095f3fbdc73144d8a2d721f56208a5\": container with ID starting with 8849ff46b633c1f9948870fe9185cb1456095f3fbdc73144d8a2d721f56208a5 not found: ID does not exist"
Oct 30 22:32:24.613113 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.613083    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b-plugins-config-volume\") pod \"f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b\" (UID: \"f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b\") "
Oct 30 22:32:24.613357 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.613168    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b-kube-api-access-54sqw\" (UniqueName: \"kubernetes.io/projected/f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b-kube-api-access-54sqw\") pod \"f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b\" (UID: \"f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b\") "
Oct 30 22:32:24.613587 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.613556    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b-plugins-config-volume" pod "f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b" (UID: "f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:24.613863 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.613846    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:24.615591 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.615562    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b-kube-api-access-54sqw" pod "f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b" (UID: "f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b"). InnerVolumeSpecName "kube-api-access-54sqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:24.715502 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.715419    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54sqw\" (UniqueName: \"kubernetes.io/projected/f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b-kube-api-access-54sqw\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:24.974352 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:24.974258    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"]
Oct 30 22:32:25.024792 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:25.024757    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3bbd9b-55df5d64c-9bpr5"]
Oct 30 22:32:25.399263 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:25.399225    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b" path="/var/lib/kubelet/pods/f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b/volumes"
Oct 30 22:32:28.201876 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:32:28.201836    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:32:28.504869 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:28.504787    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"
Oct 30 22:32:28.504869 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:28.504868    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"
Oct 30 22:32:29.912152 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:29.912104    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:32:29.917009 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:29.916986    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:32:38.633996 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:38.633960    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"]
Oct 30 22:32:38.634657 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:38.634639    4553 cpu_manager.go:414] "containerMap: removing container" podUID="f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b" containerName="payload-pre-processing"
Oct 30 22:32:38.634728 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:38.634660    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b" containerName="payload-pre-processing"
Oct 30 22:32:38.634795 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:38.634785    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="f81bb7a1-33d0-4a2b-9a42-067f2bf3ba2b" containerName="payload-pre-processing"
Oct 30 22:32:38.639975 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:38.639951    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"
Oct 30 22:32:38.694051 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:38.694017    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"]
Oct 30 22:32:38.696904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:38.696875    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98hs7\" (UniqueName: \"kubernetes.io/projected/7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932-kube-api-access-98hs7\") pod \"payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq\" (UID: \"7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"
Oct 30 22:32:38.697063 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:38.697014    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq\" (UID: \"7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"
Oct 30 22:32:38.740917 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:38.740881    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-6c788d\""
Oct 30 22:32:38.741088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:38.740949    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-6c788d-dockercfg-fw5vz\""
Oct 30 22:32:38.798364 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:38.798272    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98hs7\" (UniqueName: \"kubernetes.io/projected/7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932-kube-api-access-98hs7\") pod \"payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq\" (UID: \"7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"
Oct 30 22:32:38.798573 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:38.798429    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq\" (UID: \"7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"
Oct 30 22:32:38.801532 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:38.801104    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq\" (UID: \"7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"
Oct 30 22:32:38.885469 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:38.884578    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98hs7\" (UniqueName: \"kubernetes.io/projected/7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932-kube-api-access-98hs7\") pod \"payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq\" (UID: \"7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"
Oct 30 22:32:38.955022 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:38.954989    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"
Oct 30 22:32:39.353201 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:39.353168    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"]
Oct 30 22:32:39.354420 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:32:39.354389    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d3bd36e_4828_4ea5_ac71_d8f8f1d0b932.slice/crio-ecb9261a435dcb2857624b261fa61bdc22320e7f2b3048c191512b5517dc719c WatchSource:0}: Error finding container ecb9261a435dcb2857624b261fa61bdc22320e7f2b3048c191512b5517dc719c: Status 404 returned error can't find the container with id ecb9261a435dcb2857624b261fa61bdc22320e7f2b3048c191512b5517dc719c
Oct 30 22:32:39.649957 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:39.649856    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq" event={"ID":"7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932","Type":"ContainerStarted","Data":"55a2e1bd9d555415579910511eb9f17bc3dd425cd76995fa1ec9fd911c6eb7ac"}
Oct 30 22:32:39.649957 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:39.649914    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq" event={"ID":"7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932","Type":"ContainerStarted","Data":"ecb9261a435dcb2857624b261fa61bdc22320e7f2b3048c191512b5517dc719c"}
Oct 30 22:32:39.649957 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:39.649951    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"
Oct 30 22:32:39.764489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:39.764423    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq" podStartSLOduration=1.764408888 podStartE2EDuration="1.764408888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:39.763942077 +0000 UTC m=+3010.498305109" watchObservedRunningTime="2026-10-30 22:32:39.764408888 +0000 UTC m=+3010.498771919"
Oct 30 22:32:42.530654 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.530617    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"]
Oct 30 22:32:42.531089 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.530832    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d" podUID="9060a9ed-94f2-400e-97c4-e3fa723424d1" containerName="maas-api" containerID="cri-o://029e2dd5a2de069591c59831d1e172f4f533bd287f411b81be0d54f9ee355c13" gracePeriod=30
Oct 30 22:32:42.674356 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.674325    4553 generic.go:356] "Generic (PLEG): container finished" podID="9060a9ed-94f2-400e-97c4-e3fa723424d1" containerID="029e2dd5a2de069591c59831d1e172f4f533bd287f411b81be0d54f9ee355c13" exitCode=0
Oct 30 22:32:42.674569 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.674399    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d" event={"ID":"9060a9ed-94f2-400e-97c4-e3fa723424d1","Type":"ContainerDied","Data":"029e2dd5a2de069591c59831d1e172f4f533bd287f411b81be0d54f9ee355c13"}
Oct 30 22:32:42.834954 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.834930    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:32:42.852294 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.852265    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9060a9ed-94f2-400e-97c4-e3fa723424d1-kube-api-access-9hfg5\" (UniqueName: \"kubernetes.io/projected/9060a9ed-94f2-400e-97c4-e3fa723424d1-kube-api-access-9hfg5\") pod \"9060a9ed-94f2-400e-97c4-e3fa723424d1\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") "
Oct 30 22:32:42.852450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.852340    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-maas-api-tls\") pod \"9060a9ed-94f2-400e-97c4-e3fa723424d1\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") "
Oct 30 22:32:42.852450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.852412    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9060a9ed-94f2-400e-97c4-e3fa723424d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9060a9ed-94f2-400e-97c4-e3fa723424d1-openshift-service-ca\") pod \"9060a9ed-94f2-400e-97c4-e3fa723424d1\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") "
Oct 30 22:32:42.852581 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.852474    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-metrics-certs\") pod \"9060a9ed-94f2-400e-97c4-e3fa723424d1\" (UID: \"9060a9ed-94f2-400e-97c4-e3fa723424d1\") "
Oct 30 22:32:42.852779 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.852700    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9060a9ed-94f2-400e-97c4-e3fa723424d1-openshift-service-ca" pod "9060a9ed-94f2-400e-97c4-e3fa723424d1" (UID: "9060a9ed-94f2-400e-97c4-e3fa723424d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:42.853009 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.852986    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9060a9ed-94f2-400e-97c4-e3fa723424d1-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:42.854719 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.854694    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-metrics-certs" pod "9060a9ed-94f2-400e-97c4-e3fa723424d1" (UID: "9060a9ed-94f2-400e-97c4-e3fa723424d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:42.854801 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.854779    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-maas-api-tls" pod "9060a9ed-94f2-400e-97c4-e3fa723424d1" (UID: "9060a9ed-94f2-400e-97c4-e3fa723424d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:42.854801 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.854781    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9060a9ed-94f2-400e-97c4-e3fa723424d1-kube-api-access-9hfg5" pod "9060a9ed-94f2-400e-97c4-e3fa723424d1" (UID: "9060a9ed-94f2-400e-97c4-e3fa723424d1"). InnerVolumeSpecName "kube-api-access-9hfg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:42.958009 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.957972    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:42.958246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.958011    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hfg5\" (UniqueName: \"kubernetes.io/projected/9060a9ed-94f2-400e-97c4-e3fa723424d1-kube-api-access-9hfg5\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:42.958246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:42.958054    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9060a9ed-94f2-400e-97c4-e3fa723424d1-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:43.681670 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:43.681577    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d" event={"ID":"9060a9ed-94f2-400e-97c4-e3fa723424d1","Type":"ContainerDied","Data":"bc526846bec3f93bb8673828cfd2288db809e3aa9c41089f24a750dd9ea55d91"}
Oct 30 22:32:43.681670 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:43.681603    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"
Oct 30 22:32:43.681670 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:43.681642    4553 scope.go:122] "RemoveContainer" containerID="029e2dd5a2de069591c59831d1e172f4f533bd287f411b81be0d54f9ee355c13"
Oct 30 22:32:43.849587 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:43.849549    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"]
Oct 30 22:32:43.905735 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:43.901555    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-37a390-69d4db6c4f-2sh4d"]
Oct 30 22:32:45.394000 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:45.393964    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9060a9ed-94f2-400e-97c4-e3fa723424d1" path="/var/lib/kubelet/pods/9060a9ed-94f2-400e-97c4-e3fa723424d1/volumes"
Oct 30 22:32:47.409883 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.409847    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"]
Oct 30 22:32:47.411450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.411418    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9060a9ed-94f2-400e-97c4-e3fa723424d1" containerName="maas-api"
Oct 30 22:32:47.411450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.411450    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9060a9ed-94f2-400e-97c4-e3fa723424d1" containerName="maas-api"
Oct 30 22:32:47.411663 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.411650    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="9060a9ed-94f2-400e-97c4-e3fa723424d1" containerName="maas-api"
Oct 30 22:32:47.419451 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.419424    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"
Oct 30 22:32:47.485109 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.485068    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"]
Oct 30 22:32:47.519363 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.519323    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-b1eb3f\""
Oct 30 22:32:47.519573 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.519497    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-b1eb3f-dockercfg-tl7vj\""
Oct 30 22:32:47.542734 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.542704    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8\" (UID: \"3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"
Oct 30 22:32:47.542912 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.542761    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz269\" (UniqueName: \"kubernetes.io/projected/3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3-kube-api-access-qz269\") pod \"payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8\" (UID: \"3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"
Oct 30 22:32:47.591116 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.591077    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"]
Oct 30 22:32:47.595573 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.595549    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"
Oct 30 22:32:47.643953 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.643910    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9mbv\" (UniqueName: \"kubernetes.io/projected/289e89bb-9bee-47f8-8768-8ac8c35b47e5-kube-api-access-j9mbv\") pod \"payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq\" (UID: \"289e89bb-9bee-47f8-8768-8ac8c35b47e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"
Oct 30 22:32:47.644173 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.644122    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8\" (UID: \"3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"
Oct 30 22:32:47.644257 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.644239    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz269\" (UniqueName: \"kubernetes.io/projected/3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3-kube-api-access-qz269\") pod \"payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8\" (UID: \"3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"
Oct 30 22:32:47.644323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.644281    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289e89bb-9bee-47f8-8768-8ac8c35b47e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq\" (UID: \"289e89bb-9bee-47f8-8768-8ac8c35b47e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"
Oct 30 22:32:47.644847 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.644827    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8\" (UID: \"3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"
Oct 30 22:32:47.652355 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.652318    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"]
Oct 30 22:32:47.702668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.702595    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz269\" (UniqueName: \"kubernetes.io/projected/3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3-kube-api-access-qz269\") pod \"payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8\" (UID: \"3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"
Oct 30 22:32:47.745799 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.745760    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289e89bb-9bee-47f8-8768-8ac8c35b47e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq\" (UID: \"289e89bb-9bee-47f8-8768-8ac8c35b47e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"
Oct 30 22:32:47.745972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.745861    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9mbv\" (UniqueName: \"kubernetes.io/projected/289e89bb-9bee-47f8-8768-8ac8c35b47e5-kube-api-access-j9mbv\") pod \"payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq\" (UID: \"289e89bb-9bee-47f8-8768-8ac8c35b47e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"
Oct 30 22:32:47.746592 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.746570    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289e89bb-9bee-47f8-8768-8ac8c35b47e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq\" (UID: \"289e89bb-9bee-47f8-8768-8ac8c35b47e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"
Oct 30 22:32:47.763452 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.763423    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"
Oct 30 22:32:47.804115 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.804074    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9mbv\" (UniqueName: \"kubernetes.io/projected/289e89bb-9bee-47f8-8768-8ac8c35b47e5-kube-api-access-j9mbv\") pod \"payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq\" (UID: \"289e89bb-9bee-47f8-8768-8ac8c35b47e5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"
Oct 30 22:32:47.909825 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:47.909790    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"
Oct 30 22:32:48.148431 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:48.148399    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"]
Oct 30 22:32:48.149574 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:32:48.149535    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8a2f0e_ff56_4593_aad9_6c0a5dd48ac3.slice/crio-8ad18c0f42a701e384519017d7654a665de1e135848c00486d86673766d4d8c9 WatchSource:0}: Error finding container 8ad18c0f42a701e384519017d7654a665de1e135848c00486d86673766d4d8c9: Status 404 returned error can't find the container with id 8ad18c0f42a701e384519017d7654a665de1e135848c00486d86673766d4d8c9
Oct 30 22:32:48.291152 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:48.291099    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"]
Oct 30 22:32:48.291966 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:32:48.291935    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289e89bb_9bee_47f8_8768_8ac8c35b47e5.slice/crio-c563910daea6708c98b86da338ea9d85bc087439d781d23a36eb6a0dd3b7c285 WatchSource:0}: Error finding container c563910daea6708c98b86da338ea9d85bc087439d781d23a36eb6a0dd3b7c285: Status 404 returned error can't find the container with id c563910daea6708c98b86da338ea9d85bc087439d781d23a36eb6a0dd3b7c285
Oct 30 22:32:48.725239 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:48.725198    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq" event={"ID":"289e89bb-9bee-47f8-8768-8ac8c35b47e5","Type":"ContainerStarted","Data":"def8e973ef186f5abf22374f8931585193301a7fd01af3b6656cfc40fae9804d"}
Oct 30 22:32:48.726743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:48.726234    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq" event={"ID":"289e89bb-9bee-47f8-8768-8ac8c35b47e5","Type":"ContainerStarted","Data":"c563910daea6708c98b86da338ea9d85bc087439d781d23a36eb6a0dd3b7c285"}
Oct 30 22:32:48.726743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:48.726298    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"
Oct 30 22:32:48.731180 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:48.729258    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8" event={"ID":"3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3","Type":"ContainerStarted","Data":"4a87b50dc81c246b1b79f4949bf9e0662c49dcc88222ca114f12469c574aaccd"}
Oct 30 22:32:48.731180 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:48.729310    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8" event={"ID":"3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3","Type":"ContainerStarted","Data":"8ad18c0f42a701e384519017d7654a665de1e135848c00486d86673766d4d8c9"}
Oct 30 22:32:48.731180 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:48.729519    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"
Oct 30 22:32:48.844303 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:48.844247    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq" podStartSLOduration=1.8442316399999998 podStartE2EDuration="1.84423164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:48.840831332 +0000 UTC m=+3019.575194363" watchObservedRunningTime="2026-10-30 22:32:48.84423164 +0000 UTC m=+3019.578594761"
Oct 30 22:32:49.125771 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.125722    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8" podStartSLOduration=2.125704126 podStartE2EDuration="2.125704126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:48.969470427 +0000 UTC m=+3019.703833458" watchObservedRunningTime="2026-10-30 22:32:49.125704126 +0000 UTC m=+3019.860067156"
Oct 30 22:32:49.127179 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.127111    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"]
Oct 30 22:32:49.127348 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.127326    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m" podUID="d9fe0777-d9c7-476d-bf9c-9cd0a87394e3" containerName="maas-api" containerID="cri-o://af514ececf9d1bbfcf52a4f07b567b57fa9302b9ea84579037864b50871517a9" gracePeriod=30
Oct 30 22:32:49.447272 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.447247    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:32:49.595915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.595881    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-metrics-certs\") pod \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") "
Oct 30 22:32:49.596126 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.596043    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-kube-api-access-xgntd\" (UniqueName: \"kubernetes.io/projected/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-kube-api-access-xgntd\") pod \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") "
Oct 30 22:32:49.596251 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.596123    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-openshift-service-ca\") pod \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") "
Oct 30 22:32:49.596251 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.596199    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-maas-api-tls\") pod \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\" (UID: \"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3\") "
Oct 30 22:32:49.596996 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.596964    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-openshift-service-ca" pod "d9fe0777-d9c7-476d-bf9c-9cd0a87394e3" (UID: "d9fe0777-d9c7-476d-bf9c-9cd0a87394e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:49.601974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.600235    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-metrics-certs" pod "d9fe0777-d9c7-476d-bf9c-9cd0a87394e3" (UID: "d9fe0777-d9c7-476d-bf9c-9cd0a87394e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:49.604341 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.604309    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-kube-api-access-xgntd" pod "d9fe0777-d9c7-476d-bf9c-9cd0a87394e3" (UID: "d9fe0777-d9c7-476d-bf9c-9cd0a87394e3"). InnerVolumeSpecName "kube-api-access-xgntd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:49.604588 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.604564    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-maas-api-tls" pod "d9fe0777-d9c7-476d-bf9c-9cd0a87394e3" (UID: "d9fe0777-d9c7-476d-bf9c-9cd0a87394e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:49.700466 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.699834    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgntd\" (UniqueName: \"kubernetes.io/projected/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-kube-api-access-xgntd\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:49.700466 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.699873    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:49.700466 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.699888    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:49.700466 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.699902    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:49.739797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.739764    4553 generic.go:356] "Generic (PLEG): container finished" podID="d9fe0777-d9c7-476d-bf9c-9cd0a87394e3" containerID="af514ececf9d1bbfcf52a4f07b567b57fa9302b9ea84579037864b50871517a9" exitCode=0
Oct 30 22:32:49.740231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.739827    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"
Oct 30 22:32:49.740231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.739849    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m" event={"ID":"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3","Type":"ContainerDied","Data":"af514ececf9d1bbfcf52a4f07b567b57fa9302b9ea84579037864b50871517a9"}
Oct 30 22:32:49.740231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.739907    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m" event={"ID":"d9fe0777-d9c7-476d-bf9c-9cd0a87394e3","Type":"ContainerDied","Data":"edcf4ab57075f6e6797a8999fc246beb3f6479336ea9ab7c0e66e5b5b7bbfe6d"}
Oct 30 22:32:49.740231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.739935    4553 scope.go:122] "RemoveContainer" containerID="af514ececf9d1bbfcf52a4f07b567b57fa9302b9ea84579037864b50871517a9"
Oct 30 22:32:49.751290 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.751272    4553 scope.go:122] "RemoveContainer" containerID="af514ececf9d1bbfcf52a4f07b567b57fa9302b9ea84579037864b50871517a9"
Oct 30 22:32:49.751567 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:32:49.751548    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af514ececf9d1bbfcf52a4f07b567b57fa9302b9ea84579037864b50871517a9\": container with ID starting with af514ececf9d1bbfcf52a4f07b567b57fa9302b9ea84579037864b50871517a9 not found: ID does not exist" containerID="af514ececf9d1bbfcf52a4f07b567b57fa9302b9ea84579037864b50871517a9"
Oct 30 22:32:49.751615 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.751576    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af514ececf9d1bbfcf52a4f07b567b57fa9302b9ea84579037864b50871517a9"} err="failed to get container status \"af514ececf9d1bbfcf52a4f07b567b57fa9302b9ea84579037864b50871517a9\": rpc error: code = NotFound desc = could not find container \"af514ececf9d1bbfcf52a4f07b567b57fa9302b9ea84579037864b50871517a9\": container with ID starting with af514ececf9d1bbfcf52a4f07b567b57fa9302b9ea84579037864b50871517a9 not found: ID does not exist"
Oct 30 22:32:49.907488 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.907454    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"]
Oct 30 22:32:49.910662 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:49.910640    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3bbd9b-85d8b99454-rg42m"]
Oct 30 22:32:50.661850 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:50.661824    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"
Oct 30 22:32:51.400377 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:51.400339    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9fe0777-d9c7-476d-bf9c-9cd0a87394e3" path="/var/lib/kubelet/pods/d9fe0777-d9c7-476d-bf9c-9cd0a87394e3/volumes"
Oct 30 22:32:51.594036 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:51.593994    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"]
Oct 30 22:32:51.594320 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:51.594284    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l" podUID="7a2af516-6dad-461a-9674-8951b4231d49" containerName="istio-proxy" containerID="cri-o://cc0e2cc9c2344e134571d4d7790e6ebbf35ebe7cf8c9cd79cb7068a2e8f3bbec" gracePeriod=30
Oct 30 22:32:54.619012 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:54.618975    4553 patch_prober.go:29] interesting pod/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:32:54.619438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:54.619028    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l" podUID="7a2af516-6dad-461a-9674-8951b4231d49" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:32:55.902983 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:55.902949    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"]
Oct 30 22:32:55.903648 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:55.903632    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d9fe0777-d9c7-476d-bf9c-9cd0a87394e3" containerName="maas-api"
Oct 30 22:32:55.903690 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:55.903649    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9fe0777-d9c7-476d-bf9c-9cd0a87394e3" containerName="maas-api"
Oct 30 22:32:55.903754 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:55.903745    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9fe0777-d9c7-476d-bf9c-9cd0a87394e3" containerName="maas-api"
Oct 30 22:32:55.908623 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:55.908604    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:55.964317 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:55.964275    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"]
Oct 30 22:32:55.991580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:55.991549    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:55.991775 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:55.991619    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d352d2-8937-4ae7-9c60-2b5324fcea80-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:55.991775 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:55.991672    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:55.991852 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:55.991808    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6c6h\" (UniqueName: \"kubernetes.io/projected/b2d352d2-8937-4ae7-9c60-2b5324fcea80-kube-api-access-p6c6h\") pod \"maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:56.010423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.010396    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-b4deef-serving-cert\""
Oct 30 22:32:56.105920 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.105157    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:56.106320 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.106286    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d352d2-8937-4ae7-9c60-2b5324fcea80-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:56.106455 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.106343    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:56.106972 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:32:56.106946    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-serving-cert: secret "maas-api-e2e-shared-a-w1-b4deef-serving-cert" not found
Oct 30 22:32:56.107029 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.107013    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6c6h\" (UniqueName: \"kubernetes.io/projected/b2d352d2-8937-4ae7-9c60-2b5324fcea80-kube-api-access-p6c6h\") pod \"maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:56.107127 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.107110    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d352d2-8937-4ae7-9c60-2b5324fcea80-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:56.107535 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:32:56.107514    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-maas-api-tls podName:b2d352d2-8937-4ae7-9c60-2b5324fcea80 nodeName:}" failed. No retries permitted until 2026-10-30 22:32:56.60721902 +0000 UTC m=+3027.341582042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-maas-api-tls") pod "maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6" (UID: "b2d352d2-8937-4ae7-9c60-2b5324fcea80") : secret "maas-api-e2e-shared-a-w1-b4deef-serving-cert" not found
Oct 30 22:32:56.108568 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.108540    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:56.169592 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.169502    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6c6h\" (UniqueName: \"kubernetes.io/projected/b2d352d2-8937-4ae7-9c60-2b5324fcea80-kube-api-access-p6c6h\") pod \"maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:56.613441 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.613409    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:56.615992 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.615963    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:56.804212 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.804170    4553 generic.go:356] "Generic (PLEG): container finished" podID="7a2af516-6dad-461a-9674-8951b4231d49" containerID="cc0e2cc9c2344e134571d4d7790e6ebbf35ebe7cf8c9cd79cb7068a2e8f3bbec" exitCode=0
Oct 30 22:32:56.804422 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.804259    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l" event={"ID":"7a2af516-6dad-461a-9674-8951b4231d49","Type":"ContainerDied","Data":"cc0e2cc9c2344e134571d4d7790e6ebbf35ebe7cf8c9cd79cb7068a2e8f3bbec"}
Oct 30 22:32:56.804422 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.804367    4553 scope.go:122] "RemoveContainer" containerID="1293b95ef6bfaf6824859b78f419990577047219114765459a00efb46c2d61f7"
Oct 30 22:32:56.826422 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.826395    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:56.866965 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.866939    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:32:56.919550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.918504    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2af516-6dad-461a-9674-8951b4231d49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2af516-6dad-461a-9674-8951b4231d49-istiod-ca-cert\") pod \"7a2af516-6dad-461a-9674-8951b4231d49\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") "
Oct 30 22:32:56.919550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.918581    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-workload-socket\") pod \"7a2af516-6dad-461a-9674-8951b4231d49\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") "
Oct 30 22:32:56.919550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.918624    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2af516-6dad-461a-9674-8951b4231d49-istio-token\" (UniqueName: \"kubernetes.io/projected/7a2af516-6dad-461a-9674-8951b4231d49-istio-token\") pod \"7a2af516-6dad-461a-9674-8951b4231d49\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") "
Oct 30 22:32:56.919550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.918655    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a2af516-6dad-461a-9674-8951b4231d49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2af516-6dad-461a-9674-8951b4231d49-istio-podinfo\") pod \"7a2af516-6dad-461a-9674-8951b4231d49\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") "
Oct 30 22:32:56.919550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.918692    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-credential-socket\") pod \"7a2af516-6dad-461a-9674-8951b4231d49\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") "
Oct 30 22:32:56.919550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.918754    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-istio-envoy\") pod \"7a2af516-6dad-461a-9674-8951b4231d49\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") "
Oct 30 22:32:56.919550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.918806    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-workload-certs\") pod \"7a2af516-6dad-461a-9674-8951b4231d49\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") "
Oct 30 22:32:56.919550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.918846    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2af516-6dad-461a-9674-8951b4231d49-kube-api-access-5xthz\" (UniqueName: \"kubernetes.io/projected/7a2af516-6dad-461a-9674-8951b4231d49-kube-api-access-5xthz\") pod \"7a2af516-6dad-461a-9674-8951b4231d49\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") "
Oct 30 22:32:56.919550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.918889    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-istio-data\") pod \"7a2af516-6dad-461a-9674-8951b4231d49\" (UID: \"7a2af516-6dad-461a-9674-8951b4231d49\") "
Oct 30 22:32:56.920437 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.920270    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-istio-data" pod "7a2af516-6dad-461a-9674-8951b4231d49" (UID: "7a2af516-6dad-461a-9674-8951b4231d49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:56.920760 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.920543    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-workload-certs" pod "7a2af516-6dad-461a-9674-8951b4231d49" (UID: "7a2af516-6dad-461a-9674-8951b4231d49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:56.921401 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.921040    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-workload-socket" pod "7a2af516-6dad-461a-9674-8951b4231d49" (UID: "7a2af516-6dad-461a-9674-8951b4231d49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:56.921401 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.921065    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-credential-socket" pod "7a2af516-6dad-461a-9674-8951b4231d49" (UID: "7a2af516-6dad-461a-9674-8951b4231d49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:56.921401 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.921386    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2af516-6dad-461a-9674-8951b4231d49-istiod-ca-cert" pod "7a2af516-6dad-461a-9674-8951b4231d49" (UID: "7a2af516-6dad-461a-9674-8951b4231d49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:56.923403 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.923304    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a2af516-6dad-461a-9674-8951b4231d49-istio-podinfo" pod "7a2af516-6dad-461a-9674-8951b4231d49" (UID: "7a2af516-6dad-461a-9674-8951b4231d49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:32:56.923403 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.923385    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2af516-6dad-461a-9674-8951b4231d49-kube-api-access-5xthz" pod "7a2af516-6dad-461a-9674-8951b4231d49" (UID: "7a2af516-6dad-461a-9674-8951b4231d49"). InnerVolumeSpecName "kube-api-access-5xthz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:56.924702 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.924679    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-istio-envoy" pod "7a2af516-6dad-461a-9674-8951b4231d49" (UID: "7a2af516-6dad-461a-9674-8951b4231d49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:56.924835 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:56.924815    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2af516-6dad-461a-9674-8951b4231d49-istio-token" pod "7a2af516-6dad-461a-9674-8951b4231d49" (UID: "7a2af516-6dad-461a-9674-8951b4231d49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:57.020511 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:57.020474    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a2af516-6dad-461a-9674-8951b4231d49-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:57.020511 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:57.020504    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a2af516-6dad-461a-9674-8951b4231d49-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:57.020511 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:57.020514    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:57.020511 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:57.020522    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:57.020806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:57.020531    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:57.020806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:57.020540    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xthz\" (UniqueName: \"kubernetes.io/projected/7a2af516-6dad-461a-9674-8951b4231d49-kube-api-access-5xthz\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:57.020806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:57.020550    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:57.020806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:57.020558    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a2af516-6dad-461a-9674-8951b4231d49-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:57.020806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:57.020566    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a2af516-6dad-461a-9674-8951b4231d49-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:57.217791 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:57.217760    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"]
Oct 30 22:32:57.219038 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:32:57.218997    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d352d2_8937_4ae7_9c60_2b5324fcea80.slice/crio-71c115be4375fab669a6300c192cb6e46e657b276673c755117722fdae7d6f50 WatchSource:0}: Error finding container 71c115be4375fab669a6300c192cb6e46e657b276673c755117722fdae7d6f50: Status 404 returned error can't find the container with id 71c115be4375fab669a6300c192cb6e46e657b276673c755117722fdae7d6f50
Oct 30 22:32:57.812504 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:57.812411    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"
Oct 30 22:32:57.812504 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:57.812425    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l" event={"ID":"7a2af516-6dad-461a-9674-8951b4231d49","Type":"ContainerDied","Data":"f99757da4d90836061e3163c2c5a6cf5c22d2f867272d6b71589ab3fd867bcdf"}
Oct 30 22:32:57.812504 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:57.812482    4553 scope.go:122] "RemoveContainer" containerID="cc0e2cc9c2344e134571d4d7790e6ebbf35ebe7cf8c9cd79cb7068a2e8f3bbec"
Oct 30 22:32:57.813897 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:57.813842    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6" event={"ID":"b2d352d2-8937-4ae7-9c60-2b5324fcea80","Type":"ContainerStarted","Data":"71c115be4375fab669a6300c192cb6e46e657b276673c755117722fdae7d6f50"}
Oct 30 22:32:57.979837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:57.979804    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"]
Oct 30 22:32:58.030457 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:58.030427    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-37a390-openshift-default-cb9fdf5f7-wl98l"]
Oct 30 22:32:58.212375 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:32:58.212337    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:32:58.824381 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:58.824345    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6" event={"ID":"b2d352d2-8937-4ae7-9c60-2b5324fcea80","Type":"ContainerStarted","Data":"2261368e151727a60e9ca2251ab0374172588d6c5880274ceb6e0cc596c1e607"}
Oct 30 22:32:58.824609 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:58.824457    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:32:58.943989 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:58.943938    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6" podStartSLOduration=3.097845322 podStartE2EDuration="3.943918001s" totalImagesPullingTime="846.072679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:58.940703502 +0000 UTC m=+3029.675066544" watchObservedRunningTime="2026-10-30 22:32:58.943918001 +0000 UTC m=+3029.678281035"
Oct 30 22:32:59.394578 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:59.394550    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2af516-6dad-461a-9674-8951b4231d49" path="/var/lib/kubelet/pods/7a2af516-6dad-461a-9674-8951b4231d49/volumes"
Oct 30 22:32:59.742907 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:59.742826    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"
Oct 30 22:32:59.743066 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:32:59.742955    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"
Oct 30 22:33:04.836110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:04.836082    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:33:08.057358 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:08.057323    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"]
Oct 30 22:33:08.057828 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:08.057544    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44" podUID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" containerName="istio-proxy" containerID="cri-o://2208d28a03a537b9bd63ada2fce778c6e8d75a9be00482ecb8ed2e4446bed13d" gracePeriod=30
Oct 30 22:33:13.395150 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.395095    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:33:13.477483 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.477449    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-envoy\") pod \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") "
Oct 30 22:33:13.477702 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.477557    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-workload-certs\") pod \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") "
Oct 30 22:33:13.477702 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.477587    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istiod-ca-cert\") pod \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") "
Oct 30 22:33:13.477839 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.477702    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-token\" (UniqueName: \"kubernetes.io/projected/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-token\") pod \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") "
Oct 30 22:33:13.477839 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.477808    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-data\") pod \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") "
Oct 30 22:33:13.477956 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.477861    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-workload-socket\") pod \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") "
Oct 30 22:33:13.477956 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.477876    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-workload-certs" pod "f8af1bfa-6dda-45a6-9435-1ec9b4afd530" (UID: "f8af1bfa-6dda-45a6-9435-1ec9b4afd530"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:13.477956 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.477922    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-kube-api-access-hnm85\" (UniqueName: \"kubernetes.io/projected/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-kube-api-access-hnm85\") pod \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") "
Oct 30 22:33:13.478199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.477969    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-podinfo\") pod \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") "
Oct 30 22:33:13.478199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.478056    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-data" pod "f8af1bfa-6dda-45a6-9435-1ec9b4afd530" (UID: "f8af1bfa-6dda-45a6-9435-1ec9b4afd530"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:13.478199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.478187    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-credential-socket\") pod \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\" (UID: \"f8af1bfa-6dda-45a6-9435-1ec9b4afd530\") "
Oct 30 22:33:13.478382 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.478204    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-workload-socket" pod "f8af1bfa-6dda-45a6-9435-1ec9b4afd530" (UID: "f8af1bfa-6dda-45a6-9435-1ec9b4afd530"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:13.478382 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.478083    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istiod-ca-cert" pod "f8af1bfa-6dda-45a6-9435-1ec9b4afd530" (UID: "f8af1bfa-6dda-45a6-9435-1ec9b4afd530"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:13.478509 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.478494    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-credential-socket" pod "f8af1bfa-6dda-45a6-9435-1ec9b4afd530" (UID: "f8af1bfa-6dda-45a6-9435-1ec9b4afd530"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:13.478785 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.478762    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:13.478785 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.478783    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:13.478976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.478792    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:13.478976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.478801    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:13.478976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.478811    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:13.480061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.480039    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-envoy" pod "f8af1bfa-6dda-45a6-9435-1ec9b4afd530" (UID: "f8af1bfa-6dda-45a6-9435-1ec9b4afd530"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:13.480849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.480820    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-token" pod "f8af1bfa-6dda-45a6-9435-1ec9b4afd530" (UID: "f8af1bfa-6dda-45a6-9435-1ec9b4afd530"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:13.480946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.480896    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-podinfo" pod "f8af1bfa-6dda-45a6-9435-1ec9b4afd530" (UID: "f8af1bfa-6dda-45a6-9435-1ec9b4afd530"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:13.481001 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.480983    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-kube-api-access-hnm85" pod "f8af1bfa-6dda-45a6-9435-1ec9b4afd530" (UID: "f8af1bfa-6dda-45a6-9435-1ec9b4afd530"). InnerVolumeSpecName "kube-api-access-hnm85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:13.580083 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.580033    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnm85\" (UniqueName: \"kubernetes.io/projected/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-kube-api-access-hnm85\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:13.580083 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.580074    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:13.580333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.580092    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:13.580333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.580106    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8af1bfa-6dda-45a6-9435-1ec9b4afd530-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:33:13.959814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.952626    4553 generic.go:356] "Generic (PLEG): container finished" podID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" containerID="2208d28a03a537b9bd63ada2fce778c6e8d75a9be00482ecb8ed2e4446bed13d" exitCode=0
Oct 30 22:33:13.959814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.952700    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44" event={"ID":"f8af1bfa-6dda-45a6-9435-1ec9b4afd530","Type":"ContainerDied","Data":"2208d28a03a537b9bd63ada2fce778c6e8d75a9be00482ecb8ed2e4446bed13d"}
Oct 30 22:33:13.959814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.952751    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44" event={"ID":"f8af1bfa-6dda-45a6-9435-1ec9b4afd530","Type":"ContainerDied","Data":"a05f1a01bc0c1800cf2ed71674b9d06e04d40d12603874dd3963aef2b873d432"}
Oct 30 22:33:13.959814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.952701    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"
Oct 30 22:33:13.959814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.952782    4553 scope.go:122] "RemoveContainer" containerID="2208d28a03a537b9bd63ada2fce778c6e8d75a9be00482ecb8ed2e4446bed13d"
Oct 30 22:33:13.973288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.973257    4553 scope.go:122] "RemoveContainer" containerID="53f25c23bcbec20b6c880861c6745417f308881364a7bd824e1d48fffac04ab4"
Oct 30 22:33:13.985701 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.985676    4553 scope.go:122] "RemoveContainer" containerID="2208d28a03a537b9bd63ada2fce778c6e8d75a9be00482ecb8ed2e4446bed13d"
Oct 30 22:33:13.986171 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:33:13.986120    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2208d28a03a537b9bd63ada2fce778c6e8d75a9be00482ecb8ed2e4446bed13d\": container with ID starting with 2208d28a03a537b9bd63ada2fce778c6e8d75a9be00482ecb8ed2e4446bed13d not found: ID does not exist" containerID="2208d28a03a537b9bd63ada2fce778c6e8d75a9be00482ecb8ed2e4446bed13d"
Oct 30 22:33:13.986326 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.986174    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2208d28a03a537b9bd63ada2fce778c6e8d75a9be00482ecb8ed2e4446bed13d"} err="failed to get container status \"2208d28a03a537b9bd63ada2fce778c6e8d75a9be00482ecb8ed2e4446bed13d\": rpc error: code = NotFound desc = could not find container \"2208d28a03a537b9bd63ada2fce778c6e8d75a9be00482ecb8ed2e4446bed13d\": container with ID starting with 2208d28a03a537b9bd63ada2fce778c6e8d75a9be00482ecb8ed2e4446bed13d not found: ID does not exist"
Oct 30 22:33:13.986326 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.986200    4553 scope.go:122] "RemoveContainer" containerID="53f25c23bcbec20b6c880861c6745417f308881364a7bd824e1d48fffac04ab4"
Oct 30 22:33:13.986648 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:33:13.986622    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f25c23bcbec20b6c880861c6745417f308881364a7bd824e1d48fffac04ab4\": container with ID starting with 53f25c23bcbec20b6c880861c6745417f308881364a7bd824e1d48fffac04ab4 not found: ID does not exist" containerID="53f25c23bcbec20b6c880861c6745417f308881364a7bd824e1d48fffac04ab4"
Oct 30 22:33:13.986721 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:13.986661    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f25c23bcbec20b6c880861c6745417f308881364a7bd824e1d48fffac04ab4"} err="failed to get container status \"53f25c23bcbec20b6c880861c6745417f308881364a7bd824e1d48fffac04ab4\": rpc error: code = NotFound desc = could not find container \"53f25c23bcbec20b6c880861c6745417f308881364a7bd824e1d48fffac04ab4\": container with ID starting with 53f25c23bcbec20b6c880861c6745417f308881364a7bd824e1d48fffac04ab4 not found: ID does not exist"
Oct 30 22:33:14.145158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:14.145022    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"]
Oct 30 22:33:14.182039 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:14.182006    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-3bbd9b-openshift-default-65b67b949c-htr44"]
Oct 30 22:33:15.394001 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:15.393957    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" path="/var/lib/kubelet/pods/f8af1bfa-6dda-45a6-9435-1ec9b4afd530/volumes"
Oct 30 22:33:18.982749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:18.982714    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"]
Oct 30 22:33:18.983695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:18.983669    4553 cpu_manager.go:414] "containerMap: removing container" podUID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" containerName="istio-proxy"
Oct 30 22:33:18.983695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:18.983695    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" containerName="istio-proxy"
Oct 30 22:33:18.983883 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:18.983733    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7a2af516-6dad-461a-9674-8951b4231d49" containerName="istio-proxy"
Oct 30 22:33:18.983883 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:18.983741    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2af516-6dad-461a-9674-8951b4231d49" containerName="istio-proxy"
Oct 30 22:33:18.983883 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:18.983763    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7a2af516-6dad-461a-9674-8951b4231d49" containerName="istio-proxy"
Oct 30 22:33:18.983883 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:18.983771    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2af516-6dad-461a-9674-8951b4231d49" containerName="istio-proxy"
Oct 30 22:33:18.984098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:18.983951    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" containerName="istio-proxy"
Oct 30 22:33:18.984098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:18.983968    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" containerName="istio-proxy"
Oct 30 22:33:18.984098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:18.983983    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2af516-6dad-461a-9674-8951b4231d49" containerName="istio-proxy"
Oct 30 22:33:18.984098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:18.983996    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2af516-6dad-461a-9674-8951b4231d49" containerName="istio-proxy"
Oct 30 22:33:18.988536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:18.988516    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.103307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.044054    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-89cf28-openshift-default-dockercfg-kfnjq\""
Oct 30 22:33:19.103307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.077572    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d99s\" (UniqueName: \"kubernetes.io/projected/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-kube-api-access-2d99s\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.103307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.077642    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-token\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.103307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.077687    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-envoy\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.103307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.077725    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-data\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.103307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.077750    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-podinfo\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.103307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.077812    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-workload-certs\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.103307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.077935    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istiod-ca-cert\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.103307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.078022    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-credential-socket\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.103307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.078051    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-workload-socket\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.103307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.100006    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"]
Oct 30 22:33:19.179814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.179773    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-token\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.180015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.179848    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-envoy\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.180015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.179889    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-data\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.180015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.179915    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-podinfo\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.180015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.179974    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-workload-certs\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.180292 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.180094    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istiod-ca-cert\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.180292 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.180241    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-credential-socket\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.180408 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.180300    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-workload-socket\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.180408 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.180335    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d99s\" (UniqueName: \"kubernetes.io/projected/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-kube-api-access-2d99s\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.180950 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.180547    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-workload-certs\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.180950 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.180815    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-data\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.181505 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.181163    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-credential-socket\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.181505 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.181260    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-workload-socket\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.181630 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.181506    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istiod-ca-cert\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.189007 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.188975    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-podinfo\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.189170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.188984    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-envoy\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.239922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.239850    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-token\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.242863 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.242829    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d99s\" (UniqueName: \"kubernetes.io/projected/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-kube-api-access-2d99s\") pod \"e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") " pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.303392 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.303352    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:19.758799 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.758767    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"]
Oct 30 22:33:19.760164 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:33:19.760122    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d3cb4a_13d1_4198_ab0e_0cf97376e53f.slice/crio-de6baa435c002b3c5e1576c9b011103f77f877eb21348386b7955c02b6428d6f WatchSource:0}: Error finding container de6baa435c002b3c5e1576c9b011103f77f877eb21348386b7955c02b6428d6f: Status 404 returned error can't find the container with id de6baa435c002b3c5e1576c9b011103f77f877eb21348386b7955c02b6428d6f
Oct 30 22:33:19.762732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.762701    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:33:19.762821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.762770    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:33:19.762821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:19.762799    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:33:20.002010 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:20.001962    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn" event={"ID":"51d3cb4a-13d1-4198-ab0e-0cf97376e53f","Type":"ContainerStarted","Data":"8f38fa8af5fcc870c9d03f8fbee64f101bc37b64b3ac9a8cb20a1838332936c5"}
Oct 30 22:33:20.002528 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:20.002074    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn" event={"ID":"51d3cb4a-13d1-4198-ab0e-0cf97376e53f","Type":"ContainerStarted","Data":"de6baa435c002b3c5e1576c9b011103f77f877eb21348386b7955c02b6428d6f"}
Oct 30 22:33:20.124429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:20.124369    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn" podStartSLOduration=2.124350624 podStartE2EDuration="2.124350624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:20.121093639 +0000 UTC m=+3050.855456671" watchObservedRunningTime="2026-10-30 22:33:20.124350624 +0000 UTC m=+3050.858713656"
Oct 30 22:33:20.304323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:20.304271    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:20.310474 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:20.310438    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:21.008714 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:21.008682    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:21.009816 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:21.009797    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:33:23.022860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:23.022769    4553 generic.go:356] "Generic (PLEG): container finished" podID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerID="a1f8da1485dd857622bce01acd471fdf203b396d3a42b55d3ab0a9bfefb2056a" exitCode=0
Oct 30 22:33:23.022860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:23.022846    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp" event={"ID":"c6e1ba27-d171-44ac-b625-2d2e34fb1481","Type":"ContainerDied","Data":"a1f8da1485dd857622bce01acd471fdf203b396d3a42b55d3ab0a9bfefb2056a"}
Oct 30 22:33:23.023489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:23.023474    4553 scope.go:122] "RemoveContainer" containerID="a1f8da1485dd857622bce01acd471fdf203b396d3a42b55d3ab0a9bfefb2056a"
Oct 30 22:33:23.024913 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:23.024885    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:33:23.025006 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:23.024951    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:33:23.025006 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:23.024982    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:33:24.044595 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:24.044564    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp" event={"ID":"c6e1ba27-d171-44ac-b625-2d2e34fb1481","Type":"ContainerStarted","Data":"abb7db58614a51bfd8b6343b285d02e6506c28aff6c399c0b56fb6751db9cb1b"}
Oct 30 22:33:24.358325 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:24.358291    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:33:24.358543 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:24.358456    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 30 22:33:24.358543 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:24.358508    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp" podUID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 30 22:33:25.358416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:25.358382    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 30 22:33:25.358862 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:25.358441    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp" podUID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 30 22:33:25.969110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:25.969070    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"]
Oct 30 22:33:25.969767 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:25.969752    4553 cpu_manager.go:414] "containerMap: removing container" podUID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" containerName="istio-proxy"
Oct 30 22:33:25.969826 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:25.969770    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8af1bfa-6dda-45a6-9435-1ec9b4afd530" containerName="istio-proxy"
Oct 30 22:33:25.973482 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:25.973464    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:26.036039 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.035633    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"]
Oct 30 22:33:26.071601 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.071553    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:26.071802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.071651    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwqwd\" (UniqueName: \"kubernetes.io/projected/944271c0-d01e-4e9c-9c48-a0b892421f5b-kube-api-access-wwqwd\") pod \"maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:26.071802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.071685    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/944271c0-d01e-4e9c-9c48-a0b892421f5b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:26.071802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.071713    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:26.079082 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.079054    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-6c788d-serving-cert\""
Oct 30 22:33:26.172828 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.172794    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:26.173039 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.172904    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwqwd\" (UniqueName: \"kubernetes.io/projected/944271c0-d01e-4e9c-9c48-a0b892421f5b-kube-api-access-wwqwd\") pod \"maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:26.173039 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.172937    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/944271c0-d01e-4e9c-9c48-a0b892421f5b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:26.173039 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:33:26.172947    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-serving-cert: secret "maas-api-e2e-shared-b-w2-6c788d-serving-cert" not found
Oct 30 22:33:26.173039 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.172975    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:26.173039 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:33:26.173024    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-maas-api-tls podName:944271c0-d01e-4e9c-9c48-a0b892421f5b nodeName:}" failed. No retries permitted until 2026-10-30 22:33:26.6730059 +0000 UTC m=+3057.407368931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-maas-api-tls") pod "maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6" (UID: "944271c0-d01e-4e9c-9c48-a0b892421f5b") : secret "maas-api-e2e-shared-b-w2-6c788d-serving-cert" not found
Oct 30 22:33:26.173603 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.173581    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/944271c0-d01e-4e9c-9c48-a0b892421f5b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:26.175632 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.175608    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:26.236040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.235962    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwqwd\" (UniqueName: \"kubernetes.io/projected/944271c0-d01e-4e9c-9c48-a0b892421f5b-kube-api-access-wwqwd\") pod \"maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:26.358947 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.358913    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 30 22:33:26.359394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.358964    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp" podUID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 30 22:33:26.556031 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.555943    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"]
Oct 30 22:33:26.560381 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.560359    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"
Oct 30 22:33:26.582987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.582957    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049256e6-63e4-4ae9-9943-5c03d0091451-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r\" (UID: \"049256e6-63e4-4ae9-9943-5c03d0091451\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"
Oct 30 22:33:26.583164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.583015    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnq9t\" (UniqueName: \"kubernetes.io/projected/049256e6-63e4-4ae9-9943-5c03d0091451-kube-api-access-mnq9t\") pod \"payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r\" (UID: \"049256e6-63e4-4ae9-9943-5c03d0091451\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"
Oct 30 22:33:26.635168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.630492    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-89cf28\""
Oct 30 22:33:26.636526 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.636501    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-89cf28-dockercfg-5g2p6\""
Oct 30 22:33:26.640641 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.640609    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"]
Oct 30 22:33:26.693036 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.693004    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049256e6-63e4-4ae9-9943-5c03d0091451-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r\" (UID: \"049256e6-63e4-4ae9-9943-5c03d0091451\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"
Oct 30 22:33:26.694224 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.694182    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnq9t\" (UniqueName: \"kubernetes.io/projected/049256e6-63e4-4ae9-9943-5c03d0091451-kube-api-access-mnq9t\") pod \"payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r\" (UID: \"049256e6-63e4-4ae9-9943-5c03d0091451\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"
Oct 30 22:33:26.694870 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.694850    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:26.695937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.695915    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049256e6-63e4-4ae9-9943-5c03d0091451-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r\" (UID: \"049256e6-63e4-4ae9-9943-5c03d0091451\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"
Oct 30 22:33:26.754216 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.731287    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:26.759496 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.759470    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnq9t\" (UniqueName: \"kubernetes.io/projected/049256e6-63e4-4ae9-9943-5c03d0091451-kube-api-access-mnq9t\") pod \"payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r\" (UID: \"049256e6-63e4-4ae9-9943-5c03d0091451\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"
Oct 30 22:33:26.876146 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.876111    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"
Oct 30 22:33:26.888007 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:26.887984    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:27.297343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:27.297318    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"]
Oct 30 22:33:27.297907 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:33:27.297881    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049256e6_63e4_4ae9_9943_5c03d0091451.slice/crio-f12ce661ed48318c0eff17d6c0cec9b362e2ffe975b7e1871a7962dd2c791b1e WatchSource:0}: Error finding container f12ce661ed48318c0eff17d6c0cec9b362e2ffe975b7e1871a7962dd2c791b1e: Status 404 returned error can't find the container with id f12ce661ed48318c0eff17d6c0cec9b362e2ffe975b7e1871a7962dd2c791b1e
Oct 30 22:33:27.349315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:27.349288    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"]
Oct 30 22:33:27.356763 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:33:27.356736    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944271c0_d01e_4e9c_9c48_a0b892421f5b.slice/crio-f6c482071c6cba5373fb726eb522c01f1c7e3518371a9b2e8f417041745fe809 WatchSource:0}: Error finding container f6c482071c6cba5373fb726eb522c01f1c7e3518371a9b2e8f417041745fe809: Status 404 returned error can't find the container with id f6c482071c6cba5373fb726eb522c01f1c7e3518371a9b2e8f417041745fe809
Oct 30 22:33:27.358908 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:27.358887    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 30 22:33:27.359016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:27.358929    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp" podUID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 30 22:33:27.359331 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:27.359313    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:33:28.084701 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:28.084653    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r" event={"ID":"049256e6-63e4-4ae9-9943-5c03d0091451","Type":"ContainerStarted","Data":"c0ab0b1cde00524edfd4f0c2ff7826f7952c46290de88120942597dc0e5fc1aa"}
Oct 30 22:33:28.084927 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:28.084716    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r" event={"ID":"049256e6-63e4-4ae9-9943-5c03d0091451","Type":"ContainerStarted","Data":"f12ce661ed48318c0eff17d6c0cec9b362e2ffe975b7e1871a7962dd2c791b1e"}
Oct 30 22:33:28.084927 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:28.084744    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"
Oct 30 22:33:28.086147 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:28.086102    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6" event={"ID":"944271c0-d01e-4e9c-9c48-a0b892421f5b","Type":"ContainerStarted","Data":"f6c482071c6cba5373fb726eb522c01f1c7e3518371a9b2e8f417041745fe809"}
Oct 30 22:33:28.202281 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:28.202224    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r" podStartSLOduration=2.202208966 podStartE2EDuration="2.202208966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:28.200389335 +0000 UTC m=+3058.934752366" watchObservedRunningTime="2026-10-30 22:33:28.202208966 +0000 UTC m=+3058.936571995"
Oct 30 22:33:28.204817 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:33:28.204789    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:33:28.359202 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:28.358920    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body=
Oct 30 22:33:28.359202 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:28.358976    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp" podUID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused"
Oct 30 22:33:29.094018 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:29.093980    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6" event={"ID":"944271c0-d01e-4e9c-9c48-a0b892421f5b","Type":"ContainerStarted","Data":"f19b43190683653f8adc91b2ca3916e64c8e54e766f2b34857831499f1cc9d1c"}
Oct 30 22:33:29.094227 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:29.094079    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:29.212842 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:29.212790    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6" podStartSLOduration=3.302727842 podStartE2EDuration="4.21277502s" totalImagesPullingTime="910.047178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:29.211378779 +0000 UTC m=+3059.945741812" watchObservedRunningTime="2026-10-30 22:33:29.21277502 +0000 UTC m=+3059.947138051"
Oct 30 22:33:29.395352 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:29.395258    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:33:29.395833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:29.395669    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:33:29.396820 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:29.396800    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:33:35.107607 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:35.107577    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:33:36.529590 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:36.529553    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"]
Oct 30 22:33:36.535486 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:36.535460    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:36.639618 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:36.639587    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-b1eb3f-serving-cert\""
Oct 30 22:33:36.642900 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:36.642872    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"]
Oct 30 22:33:36.658095 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:36.658061    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:36.658283 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:36.658202    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa41d436-167b-4135-93b5-7b090231e65b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:36.658371 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:36.658348    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:36.658516 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:36.658453    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w67z\" (UniqueName: \"kubernetes.io/projected/fa41d436-167b-4135-93b5-7b090231e65b-kube-api-access-6w67z\") pod \"maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:36.760306 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:36.760273    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:36.760528 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:36.760370    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w67z\" (UniqueName: \"kubernetes.io/projected/fa41d436-167b-4135-93b5-7b090231e65b-kube-api-access-6w67z\") pod \"maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:36.760528 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:33:36.760431    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-serving-cert: secret "maas-api-e2e-shared-b-w0-b1eb3f-serving-cert" not found
Oct 30 22:33:36.760528 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:36.760499    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:36.760528 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:33:36.760517    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-maas-api-tls podName:fa41d436-167b-4135-93b5-7b090231e65b nodeName:}" failed. No retries permitted until 2026-10-30 22:33:37.260495056 +0000 UTC m=+3067.994858085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-maas-api-tls") pod "maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl" (UID: "fa41d436-167b-4135-93b5-7b090231e65b") : secret "maas-api-e2e-shared-b-w0-b1eb3f-serving-cert" not found
Oct 30 22:33:36.760769 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:36.760571    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa41d436-167b-4135-93b5-7b090231e65b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:36.761257 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:36.761231    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa41d436-167b-4135-93b5-7b090231e65b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:36.763510 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:36.763482    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:36.821405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:36.821331    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w67z\" (UniqueName: \"kubernetes.io/projected/fa41d436-167b-4135-93b5-7b090231e65b-kube-api-access-6w67z\") pod \"maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:37.266598 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:37.266562    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:37.269283 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:37.269260    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:37.449144 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:37.449105    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:37.844786 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:37.844753    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"]
Oct 30 22:33:37.845803 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:33:37.845777    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa41d436_167b_4135_93b5_7b090231e65b.slice/crio-fcb756b5d1db126e3bd503c948b5693ac32b69314a3799650560aa4452eb40d3 WatchSource:0}: Error finding container fcb756b5d1db126e3bd503c948b5693ac32b69314a3799650560aa4452eb40d3: Status 404 returned error can't find the container with id fcb756b5d1db126e3bd503c948b5693ac32b69314a3799650560aa4452eb40d3
Oct 30 22:33:38.149984 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:38.149899    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl" event={"ID":"fa41d436-167b-4135-93b5-7b090231e65b","Type":"ContainerStarted","Data":"fcb756b5d1db126e3bd503c948b5693ac32b69314a3799650560aa4452eb40d3"}
Oct 30 22:33:39.097995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:39.097964    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"
Oct 30 22:33:39.157367 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:39.157322    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl" event={"ID":"fa41d436-167b-4135-93b5-7b090231e65b","Type":"ContainerStarted","Data":"30006317e26b1d7fe4f2eeb209daaf4138ae96a4aa7eb2dd2c3d8f5c2341534c"}
Oct 30 22:33:39.157367 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:39.157394    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:39.330536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:39.329636    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl" podStartSLOduration=2.5111672670000003 podStartE2EDuration="3.329618697s" totalImagesPullingTime="818.45143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:39.329258918 +0000 UTC m=+3070.063621949" watchObservedRunningTime="2026-10-30 22:33:39.329618697 +0000 UTC m=+3070.063981750"
Oct 30 22:33:45.170062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:33:45.170025    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:33:58.203005 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:33:58.202970    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:34:10.360386 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.360306    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"]
Oct 30 22:34:10.364965 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.364948    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.415639 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.415589    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-istio-data\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.415824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.415649    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzj55\" (UniqueName: \"kubernetes.io/projected/d8da6201-f247-4182-a143-6a880f98b175-kube-api-access-rzj55\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.415824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.415777    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-workload-socket\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.415988 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.415863    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8da6201-f247-4182-a143-6a880f98b175-istiod-ca-cert\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.415988 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.415908    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-workload-certs\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.416111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.416015    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-credential-socket\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.416111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.416083    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8da6201-f247-4182-a143-6a880f98b175-istio-token\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.416297 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.416196    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8da6201-f247-4182-a143-6a880f98b175-istio-podinfo\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.416297 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.416242    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-istio-envoy\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.417895 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.417869    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-9f7654-openshift-default-dockercfg-fh7hg\""
Oct 30 22:34:10.473091 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.473055    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"]
Oct 30 22:34:10.517249 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.517196    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8da6201-f247-4182-a143-6a880f98b175-istiod-ca-cert\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.517475 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.517268    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-workload-certs\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.517475 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.517364    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-credential-socket\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.517475 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.517431    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8da6201-f247-4182-a143-6a880f98b175-istio-token\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.517654 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.517536    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8da6201-f247-4182-a143-6a880f98b175-istio-podinfo\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.517654 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.517583    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-istio-envoy\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.517769 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.517680    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-istio-data\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.517825 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.517774    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzj55\" (UniqueName: \"kubernetes.io/projected/d8da6201-f247-4182-a143-6a880f98b175-kube-api-access-rzj55\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.517883 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.517840    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-workload-socket\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.518098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.518072    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8da6201-f247-4182-a143-6a880f98b175-istiod-ca-cert\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.518252 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.518235    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-workload-socket\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.519379 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.519325    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-workload-certs\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.519520 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.519457    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-credential-socket\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.519928 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.519901    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-istio-data\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.521398 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.521375    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-istio-envoy\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.522588 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.522568    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8da6201-f247-4182-a143-6a880f98b175-istio-podinfo\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.577026 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.576997    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8da6201-f247-4182-a143-6a880f98b175-istio-token\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.577263 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.577121    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzj55\" (UniqueName: \"kubernetes.io/projected/d8da6201-f247-4182-a143-6a880f98b175-kube-api-access-rzj55\") pod \"e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") " pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:10.678450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:10.678351    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:11.113997 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:11.113950    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"]
Oct 30 22:34:11.115228 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:34:11.115199    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8da6201_f247_4182_a143_6a880f98b175.slice/crio-d867fd48754bd87a8de7686a072d0e5018542a9b7d53b603b7676f32f54209be WatchSource:0}: Error finding container d867fd48754bd87a8de7686a072d0e5018542a9b7d53b603b7676f32f54209be: Status 404 returned error can't find the container with id d867fd48754bd87a8de7686a072d0e5018542a9b7d53b603b7676f32f54209be
Oct 30 22:34:11.118329 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:11.118297    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:34:11.118413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:11.118366    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:34:11.118413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:11.118394    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:34:11.364535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:11.364427    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb" event={"ID":"d8da6201-f247-4182-a143-6a880f98b175","Type":"ContainerStarted","Data":"60e8df7b826a53be9952143308a4038986302ccdc984113e905ac6720b17a472"}
Oct 30 22:34:11.364535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:11.364495    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb" event={"ID":"d8da6201-f247-4182-a143-6a880f98b175","Type":"ContainerStarted","Data":"d867fd48754bd87a8de7686a072d0e5018542a9b7d53b603b7676f32f54209be"}
Oct 30 22:34:11.489000 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:11.488931    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb" podStartSLOduration=1.4889075489999999 podStartE2EDuration="1.488907549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:11.483081961 +0000 UTC m=+3102.217444992" watchObservedRunningTime="2026-10-30 22:34:11.488907549 +0000 UTC m=+3102.223270580"
Oct 30 22:34:11.679256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:11.679165    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:12.685405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:12.685374    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:13.376325 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:13.376294    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:13.377518 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:13.377497    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:34:17.971376 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:17.971341    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"]
Oct 30 22:34:17.985775 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:17.985738    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"
Oct 30 22:34:18.033025 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:18.032986    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0886f20-1c73-4f9a-8f1d-63db1a2c5aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9\" (UID: \"f0886f20-1c73-4f9a-8f1d-63db1a2c5aba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"
Oct 30 22:34:18.033256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:18.033146    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68wbp\" (UniqueName: \"kubernetes.io/projected/f0886f20-1c73-4f9a-8f1d-63db1a2c5aba-kube-api-access-68wbp\") pod \"payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9\" (UID: \"f0886f20-1c73-4f9a-8f1d-63db1a2c5aba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"
Oct 30 22:34:18.034321 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:18.034295    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"]
Oct 30 22:34:18.103326 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:18.085005    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-9f7654\""
Oct 30 22:34:18.106800 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:18.086505    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-9f7654-dockercfg-95k75\""
Oct 30 22:34:18.139089 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:18.139047    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0886f20-1c73-4f9a-8f1d-63db1a2c5aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9\" (UID: \"f0886f20-1c73-4f9a-8f1d-63db1a2c5aba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"
Oct 30 22:34:18.139620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:18.139566    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68wbp\" (UniqueName: \"kubernetes.io/projected/f0886f20-1c73-4f9a-8f1d-63db1a2c5aba-kube-api-access-68wbp\") pod \"payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9\" (UID: \"f0886f20-1c73-4f9a-8f1d-63db1a2c5aba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"
Oct 30 22:34:18.140302 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:18.140276    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0886f20-1c73-4f9a-8f1d-63db1a2c5aba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9\" (UID: \"f0886f20-1c73-4f9a-8f1d-63db1a2c5aba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"
Oct 30 22:34:18.204557 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:18.204521    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68wbp\" (UniqueName: \"kubernetes.io/projected/f0886f20-1c73-4f9a-8f1d-63db1a2c5aba-kube-api-access-68wbp\") pod \"payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9\" (UID: \"f0886f20-1c73-4f9a-8f1d-63db1a2c5aba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"
Oct 30 22:34:18.303239 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:18.303124    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"
Oct 30 22:34:18.681467 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:18.681441    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"]
Oct 30 22:34:18.682022 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:34:18.681993    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0886f20_1c73_4f9a_8f1d_63db1a2c5aba.slice/crio-0f29096fa4249110f4726f200a5b9a33d7d53483e886f682a33181597b95b45b WatchSource:0}: Error finding container 0f29096fa4249110f4726f200a5b9a33d7d53483e886f682a33181597b95b45b: Status 404 returned error can't find the container with id 0f29096fa4249110f4726f200a5b9a33d7d53483e886f682a33181597b95b45b
Oct 30 22:34:19.418342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:19.418303    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9" event={"ID":"f0886f20-1c73-4f9a-8f1d-63db1a2c5aba","Type":"ContainerStarted","Data":"d11cd22791589be1515f42c4a10201920793d1216597e0246a1aec8a5a6cfd26"}
Oct 30 22:34:19.418812 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:19.418371    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9" event={"ID":"f0886f20-1c73-4f9a-8f1d-63db1a2c5aba","Type":"ContainerStarted","Data":"0f29096fa4249110f4726f200a5b9a33d7d53483e886f682a33181597b95b45b"}
Oct 30 22:34:19.418812 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:19.418507    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"
Oct 30 22:34:19.536031 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:19.535976    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9" podStartSLOduration=2.535961421 podStartE2EDuration="2.535961421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:19.534036873 +0000 UTC m=+3110.268399904" watchObservedRunningTime="2026-10-30 22:34:19.535961421 +0000 UTC m=+3110.270324451"
Oct 30 22:34:28.203694 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:34:28.203654    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:34:29.739417 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.739380    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"]
Oct 30 22:34:29.744199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.744176    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.798494 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.798459    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-9651e9-openshift-default-dockercfg-gbdjz\""
Oct 30 22:34:29.832472 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.832436    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-podinfo\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.832668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.832502    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-token\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.832668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.832635    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-workload-socket\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.832772 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.832682    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-workload-certs\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.832772 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.832736    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-envoy\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.832885 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.832865    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmzdk\" (UniqueName: \"kubernetes.io/projected/aff22a9f-32c6-4373-8d9d-62f6ce177851-kube-api-access-mmzdk\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.832941 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.832926    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff22a9f-32c6-4373-8d9d-62f6ce177851-istiod-ca-cert\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.833056 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.833041    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-credential-socket\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.833121 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.833106    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-data\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.852400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.852367    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"]
Oct 30 22:34:29.934794 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.934763    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmzdk\" (UniqueName: \"kubernetes.io/projected/aff22a9f-32c6-4373-8d9d-62f6ce177851-kube-api-access-mmzdk\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.935004 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.934819    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff22a9f-32c6-4373-8d9d-62f6ce177851-istiod-ca-cert\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.935004 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.934888    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-credential-socket\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.935004 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.934945    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-data\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.935230 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.935048    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-podinfo\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.935230 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.935102    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-token\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.935230 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.935226    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-workload-socket\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.935413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.935263    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-workload-certs\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.935413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.935287    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-envoy\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.935536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.935442    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-credential-socket\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.935660 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.935632    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-workload-socket\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.935754 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.935658    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-workload-certs\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.935874 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.935824    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-data\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.936073 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.936042    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff22a9f-32c6-4373-8d9d-62f6ce177851-istiod-ca-cert\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.939043 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.939002    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-envoy\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.939443 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.939421    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-podinfo\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.994004 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.993919    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-token\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:29.994380 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:29.994361    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmzdk\" (UniqueName: \"kubernetes.io/projected/aff22a9f-32c6-4373-8d9d-62f6ce177851-kube-api-access-mmzdk\") pod \"e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") " pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:30.058768 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:30.058726    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:30.428829 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:30.428798    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"
Oct 30 22:34:30.504898 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:30.504867    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"]
Oct 30 22:34:30.505770 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:34:30.505740    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff22a9f_32c6_4373_8d9d_62f6ce177851.slice/crio-45b0f7f59716806a78f3ce3ebfaf4a5912ab39b9be67e843313707451a3d52dc WatchSource:0}: Error finding container 45b0f7f59716806a78f3ce3ebfaf4a5912ab39b9be67e843313707451a3d52dc: Status 404 returned error can't find the container with id 45b0f7f59716806a78f3ce3ebfaf4a5912ab39b9be67e843313707451a3d52dc
Oct 30 22:34:30.508426 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:30.508394    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:34:30.508508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:30.508461    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:34:30.508508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:30.508490    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:34:31.500223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:31.500178    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv" event={"ID":"aff22a9f-32c6-4373-8d9d-62f6ce177851","Type":"ContainerStarted","Data":"02144e37d2ad8dce8951e78a398c810f112810d14266d2e9dbc15cb2bccfad8f"}
Oct 30 22:34:31.500726 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:31.500237    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv" event={"ID":"aff22a9f-32c6-4373-8d9d-62f6ce177851","Type":"ContainerStarted","Data":"45b0f7f59716806a78f3ce3ebfaf4a5912ab39b9be67e843313707451a3d52dc"}
Oct 30 22:34:31.670994 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:31.670938    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv" podStartSLOduration=2.6709231190000002 podStartE2EDuration="2.670923119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:31.669091587 +0000 UTC m=+3122.403454619" watchObservedRunningTime="2026-10-30 22:34:31.670923119 +0000 UTC m=+3122.405286150"
Oct 30 22:34:32.059118 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:32.059082    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:32.064965 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:32.064942    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:32.507171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:32.507103    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:32.508047 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:32.508028    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:34:44.622304 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:44.622273    4553 generic.go:356] "Generic (PLEG): container finished" podID="51d3cb4a-13d1-4198-ab0e-0cf97376e53f" containerID="8f38fa8af5fcc870c9d03f8fbee64f101bc37b64b3ac9a8cb20a1838332936c5" exitCode=0
Oct 30 22:34:44.622780 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:44.622350    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn" event={"ID":"51d3cb4a-13d1-4198-ab0e-0cf97376e53f","Type":"ContainerDied","Data":"8f38fa8af5fcc870c9d03f8fbee64f101bc37b64b3ac9a8cb20a1838332936c5"}
Oct 30 22:34:44.622837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:44.622828    4553 scope.go:122] "RemoveContainer" containerID="8f38fa8af5fcc870c9d03f8fbee64f101bc37b64b3ac9a8cb20a1838332936c5"
Oct 30 22:34:44.624568 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:44.624538    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:34:44.624642 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:44.624606    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:34:44.624684 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:44.624642    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:34:45.642195 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:45.642162    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn" event={"ID":"51d3cb4a-13d1-4198-ab0e-0cf97376e53f","Type":"ContainerStarted","Data":"507c3f28caf6a2ac3426785fdb09c82de19d45ccb36d0753d1d3098b23dbe300"}
Oct 30 22:34:46.303792 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:46.303753    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:34:46.303994 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:46.303975    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:34:46.304041 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:46.304020    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn" podUID="51d3cb4a-13d1-4198-ab0e-0cf97376e53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 30 22:34:47.304464 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:47.304424    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:34:47.304877 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:47.304482    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn" podUID="51d3cb4a-13d1-4198-ab0e-0cf97376e53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 30 22:34:48.303966 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:48.303931    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:34:48.304241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:48.303985    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn" podUID="51d3cb4a-13d1-4198-ab0e-0cf97376e53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 30 22:34:49.304689 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:49.304657    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:34:49.305102 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:49.304706    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn" podUID="51d3cb4a-13d1-4198-ab0e-0cf97376e53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 30 22:34:50.163174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:50.160013    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"]
Oct 30 22:34:50.214167 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:50.212618    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"
Oct 30 22:34:50.225255 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:50.224892    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"]
Oct 30 22:34:50.266383 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:50.266253    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-9651e9\""
Oct 30 22:34:50.291065 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:50.291033    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da90d52a-d740-441b-a300-209ba4d63323-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t\" (UID: \"da90d52a-d740-441b-a300-209ba4d63323\") " pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"
Oct 30 22:34:50.291275 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:50.291167    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnlwl\" (UniqueName: \"kubernetes.io/projected/da90d52a-d740-441b-a300-209ba4d63323-kube-api-access-lnlwl\") pod \"payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t\" (UID: \"da90d52a-d740-441b-a300-209ba4d63323\") " pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"
Oct 30 22:34:50.303718 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:50.303685    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:34:50.303873 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:50.303759    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn" podUID="51d3cb4a-13d1-4198-ab0e-0cf97376e53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Oct 30 22:34:50.320458 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:50.320417    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-9651e9-dockercfg-4p5bv\""
Oct 30 22:34:50.392047 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:50.392016    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da90d52a-d740-441b-a300-209ba4d63323-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t\" (UID: \"da90d52a-d740-441b-a300-209ba4d63323\") " pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"
Oct 30 22:34:50.392331 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:50.392296    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnlwl\" (UniqueName: \"kubernetes.io/projected/da90d52a-d740-441b-a300-209ba4d63323-kube-api-access-lnlwl\") pod \"payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t\" (UID: \"da90d52a-d740-441b-a300-209ba4d63323\") " pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"
Oct 30 22:34:50.392794 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:50.392768    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da90d52a-d740-441b-a300-209ba4d63323-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t\" (UID: \"da90d52a-d740-441b-a300-209ba4d63323\") " pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"
Oct 30 22:34:50.450695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:50.450621    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnlwl\" (UniqueName: \"kubernetes.io/projected/da90d52a-d740-441b-a300-209ba4d63323-kube-api-access-lnlwl\") pod \"payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t\" (UID: \"da90d52a-d740-441b-a300-209ba4d63323\") " pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"
Oct 30 22:34:50.543377 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:50.543344    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"
Oct 30 22:34:50.923973 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:50.923949    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"]
Oct 30 22:34:50.927162 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:34:50.927105    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda90d52a_d740_441b_a300_209ba4d63323.slice/crio-f8526491f47daf5a17ec9b41c4150e78f7f131cdac54a687528d933e9a79008d WatchSource:0}: Error finding container f8526491f47daf5a17ec9b41c4150e78f7f131cdac54a687528d933e9a79008d: Status 404 returned error can't find the container with id f8526491f47daf5a17ec9b41c4150e78f7f131cdac54a687528d933e9a79008d
Oct 30 22:34:51.009332 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:51.009198    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:34:51.308972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:51.308892    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:34:51.686927 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:51.686887    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t" event={"ID":"da90d52a-d740-441b-a300-209ba4d63323","Type":"ContainerStarted","Data":"cee9f97bcf1eec8823b7f7b2706abf1d491352beda04ae3534ba20e09d5c693f"}
Oct 30 22:34:51.687472 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:51.686941    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t" event={"ID":"da90d52a-d740-441b-a300-209ba4d63323","Type":"ContainerStarted","Data":"f8526491f47daf5a17ec9b41c4150e78f7f131cdac54a687528d933e9a79008d"}
Oct 30 22:34:51.687472 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:51.687113    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"
Oct 30 22:34:51.688286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:51.688270    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:34:51.803552 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:34:51.803503    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t" podStartSLOduration=1.80348832 podStartE2EDuration="1.80348832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:51.801517322 +0000 UTC m=+3142.535880357" watchObservedRunningTime="2026-10-30 22:34:51.80348832 +0000 UTC m=+3142.537851350"
Oct 30 22:34:58.202710 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:34:58.202674    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:35:02.697237 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:02.697201    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"
Oct 30 22:35:28.202612 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:35:28.202574    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:35:41.324897 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.324864    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"]
Oct 30 22:35:41.330178 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.330145    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.395791 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.395761    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"]
Oct 30 22:35:41.432166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.432085    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-e90906\"/\"kube-root-ca.crt\""
Oct 30 22:35:41.432166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.432092    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-e90906\"/\"default-dockercfg-b5d84\""
Oct 30 22:35:41.433687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.433663    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-e90906\"/\"auth-test-model-e90906-kserve-self-signed-certs\""
Oct 30 22:35:41.433762 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.433663    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-e90906\"/\"openshift-service-ca.crt\""
Oct 30 22:35:41.434363 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.434340    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j84x\" (UniqueName: \"kubernetes.io/projected/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-kube-api-access-8j84x\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.434426 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.434412    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-tmp-dir\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.434564 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.434538    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-model-cache\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.434616 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.434599    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-home\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.434807 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.434790    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-tls-certs\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.434923 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.434908    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-dshm\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.537979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.536756    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-dshm\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.537979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.536906    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j84x\" (UniqueName: \"kubernetes.io/projected/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-kube-api-access-8j84x\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.537979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.536992    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-tmp-dir\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.537979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.537037    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-model-cache\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.537979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.537085    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-home\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.537979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.537232    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-tls-certs\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.537979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.537838    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-tmp-dir\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.537979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.537975    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-home\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.538627 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.538012    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-model-cache\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.540421 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.540395    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-dshm\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.540667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.540643    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-tls-certs\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.598154 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.598110    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j84x\" (UniqueName: \"kubernetes.io/projected/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-kube-api-access-8j84x\") pod \"auth-test-model-e90906-kserve-7b6c66d68f-rw24c\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") " pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:41.646934 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:41.646903    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:42.028543 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:42.028510    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"]
Oct 30 22:35:42.029041 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:35:42.029015    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc8615c4_fde1_43e8_b2f0_bae5a77b59b0.slice/crio-f503640e99f0a76a5de5e01193e91b2a8a3192dc03536cd68dc2c2537b107ce0 WatchSource:0}: Error finding container f503640e99f0a76a5de5e01193e91b2a8a3192dc03536cd68dc2c2537b107ce0: Status 404 returned error can't find the container with id f503640e99f0a76a5de5e01193e91b2a8a3192dc03536cd68dc2c2537b107ce0
Oct 30 22:35:42.062190 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:42.062124    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c" event={"ID":"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0","Type":"ContainerStarted","Data":"f503640e99f0a76a5de5e01193e91b2a8a3192dc03536cd68dc2c2537b107ce0"}
Oct 30 22:35:43.070898 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:43.070860    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c" event={"ID":"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0","Type":"ContainerStarted","Data":"66d577cf23c050b395579c305fd55f99b7a2c0f7e50241d8dea958b5e414c2e4"}
Oct 30 22:35:43.189506 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:43.189453    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c" podStartSLOduration=2.189438549 podStartE2EDuration="2.189438549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:43.18724011 +0000 UTC m=+3193.921603144" watchObservedRunningTime="2026-10-30 22:35:43.189438549 +0000 UTC m=+3193.923801577"
Oct 30 22:35:43.378111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:43.378073    4553 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:35:43.378370 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:43.378182    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb" podUID="d8da6201-f247-4182-a143-6a880f98b175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 22:35:43.647186 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:43.647083    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:44.079362 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:44.079271    4553 generic.go:356] "Generic (PLEG): container finished" podID="d8da6201-f247-4182-a143-6a880f98b175" containerID="60e8df7b826a53be9952143308a4038986302ccdc984113e905ac6720b17a472" exitCode=0
Oct 30 22:35:44.079362 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:44.079347    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb" event={"ID":"d8da6201-f247-4182-a143-6a880f98b175","Type":"ContainerDied","Data":"60e8df7b826a53be9952143308a4038986302ccdc984113e905ac6720b17a472"}
Oct 30 22:35:44.079908 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:44.079891    4553 scope.go:122] "RemoveContainer" containerID="60e8df7b826a53be9952143308a4038986302ccdc984113e905ac6720b17a472"
Oct 30 22:35:44.081390 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:44.081366    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:35:44.081476 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:44.081430    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:35:44.081476 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:44.081463    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:35:45.089560 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:45.089519    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb" event={"ID":"d8da6201-f247-4182-a143-6a880f98b175","Type":"ContainerStarted","Data":"eb791c5ae4268f4a917b3efe2db8399aba20ae4063af6c1fe930f154f78ea033"}
Oct 30 22:35:45.679583 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:45.679547    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:35:45.679806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:45.679758    4553 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:35:45.679889 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:45.679814    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb" podUID="d8da6201-f247-4182-a143-6a880f98b175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 22:35:46.679334 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:46.679296    4553 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:35:46.679746 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:46.679358    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb" podUID="d8da6201-f247-4182-a143-6a880f98b175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 22:35:47.678744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:47.678713    4553 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:35:47.678932 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:47.678766    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb" podUID="d8da6201-f247-4182-a143-6a880f98b175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 22:35:48.678727 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:48.678685    4553 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:35:48.679205 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:48.678744    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb" podUID="d8da6201-f247-4182-a143-6a880f98b175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 22:35:49.681809 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:49.681761    4553 patch_prober.go:29] interesting pod/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body=
Oct 30 22:35:49.682445 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:49.682414    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb" podUID="d8da6201-f247-4182-a143-6a880f98b175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused"
Oct 30 22:35:50.685080 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:50.685043    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:35:50.685520 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:50.685374    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:35:50.686191 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:50.686172    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:35:51.647307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:51.647273    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:51.660494 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:51.660469    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:52.172796 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:35:52.172768    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:35:58.202868 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:35:58.202825    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:36:00.001820 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.001786    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"]
Oct 30 22:36:00.006373 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.006354    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"
Oct 30 22:36:00.059362 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.059329    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-0e450a\""
Oct 30 22:36:00.059546 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.059383    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-0e450a-dockercfg-vmfwr\""
Oct 30 22:36:00.060563 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.060537    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"]
Oct 30 22:36:00.090754 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.090712    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daf9bde1-9416-46d4-89ed-e81e287a923f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt\" (UID: \"daf9bde1-9416-46d4-89ed-e81e287a923f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"
Oct 30 22:36:00.091026 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.091007    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpj46\" (UniqueName: \"kubernetes.io/projected/daf9bde1-9416-46d4-89ed-e81e287a923f-kube-api-access-rpj46\") pod \"payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt\" (UID: \"daf9bde1-9416-46d4-89ed-e81e287a923f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"
Oct 30 22:36:00.192214 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.188204    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"]
Oct 30 22:36:00.192717 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.192688    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daf9bde1-9416-46d4-89ed-e81e287a923f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt\" (UID: \"daf9bde1-9416-46d4-89ed-e81e287a923f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"
Oct 30 22:36:00.192916 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.192897    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpj46\" (UniqueName: \"kubernetes.io/projected/daf9bde1-9416-46d4-89ed-e81e287a923f-kube-api-access-rpj46\") pod \"payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt\" (UID: \"daf9bde1-9416-46d4-89ed-e81e287a923f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"
Oct 30 22:36:00.193565 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.193535    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daf9bde1-9416-46d4-89ed-e81e287a923f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt\" (UID: \"daf9bde1-9416-46d4-89ed-e81e287a923f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"
Oct 30 22:36:00.193914 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.193897    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"
Oct 30 22:36:00.214120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.213768    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"]
Oct 30 22:36:00.250611 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.250584    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpj46\" (UniqueName: \"kubernetes.io/projected/daf9bde1-9416-46d4-89ed-e81e287a923f-kube-api-access-rpj46\") pod \"payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt\" (UID: \"daf9bde1-9416-46d4-89ed-e81e287a923f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"
Oct 30 22:36:00.294489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.294400    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af38adcc-b4d4-4c9b-838e-e2f34115eb95-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4\" (UID: \"af38adcc-b4d4-4c9b-838e-e2f34115eb95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"
Oct 30 22:36:00.294655 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.294619    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvfbw\" (UniqueName: \"kubernetes.io/projected/af38adcc-b4d4-4c9b-838e-e2f34115eb95-kube-api-access-xvfbw\") pod \"payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4\" (UID: \"af38adcc-b4d4-4c9b-838e-e2f34115eb95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"
Oct 30 22:36:00.320972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.320931    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"
Oct 30 22:36:00.396548 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.396502    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af38adcc-b4d4-4c9b-838e-e2f34115eb95-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4\" (UID: \"af38adcc-b4d4-4c9b-838e-e2f34115eb95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"
Oct 30 22:36:00.396765 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.396745    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvfbw\" (UniqueName: \"kubernetes.io/projected/af38adcc-b4d4-4c9b-838e-e2f34115eb95-kube-api-access-xvfbw\") pod \"payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4\" (UID: \"af38adcc-b4d4-4c9b-838e-e2f34115eb95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"
Oct 30 22:36:00.400533 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.397547    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af38adcc-b4d4-4c9b-838e-e2f34115eb95-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4\" (UID: \"af38adcc-b4d4-4c9b-838e-e2f34115eb95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"
Oct 30 22:36:00.454886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.454850    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvfbw\" (UniqueName: \"kubernetes.io/projected/af38adcc-b4d4-4c9b-838e-e2f34115eb95-kube-api-access-xvfbw\") pod \"payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4\" (UID: \"af38adcc-b4d4-4c9b-838e-e2f34115eb95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"
Oct 30 22:36:00.513353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.513319    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"
Oct 30 22:36:00.702459 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.702431    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"]
Oct 30 22:36:00.703264 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:36:00.703236    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf9bde1_9416_46d4_89ed_e81e287a923f.slice/crio-2444259b8e26d05f02c39938695b8d26ccab375ec35123301a343e037d94fd32 WatchSource:0}: Error finding container 2444259b8e26d05f02c39938695b8d26ccab375ec35123301a343e037d94fd32: Status 404 returned error can't find the container with id 2444259b8e26d05f02c39938695b8d26ccab375ec35123301a343e037d94fd32
Oct 30 22:36:00.892943 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:36:00.892891    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf38adcc_b4d4_4c9b_838e_e2f34115eb95.slice/crio-c336aae85049b736968efdd76a544a8f0675b27595b082994156c9ab993bd503 WatchSource:0}: Error finding container c336aae85049b736968efdd76a544a8f0675b27595b082994156c9ab993bd503: Status 404 returned error can't find the container with id c336aae85049b736968efdd76a544a8f0675b27595b082994156c9ab993bd503
Oct 30 22:36:00.893083 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:00.892987    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"]
Oct 30 22:36:01.228170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:01.228039    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4" event={"ID":"af38adcc-b4d4-4c9b-838e-e2f34115eb95","Type":"ContainerStarted","Data":"017d1e85a430cb12dc5ca6c1bb89ba2fe6ba674fec801911135ce2ec01a28519"}
Oct 30 22:36:01.228170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:01.228101    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4" event={"ID":"af38adcc-b4d4-4c9b-838e-e2f34115eb95","Type":"ContainerStarted","Data":"c336aae85049b736968efdd76a544a8f0675b27595b082994156c9ab993bd503"}
Oct 30 22:36:01.228713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:01.228170    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"
Oct 30 22:36:01.229952 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:01.229926    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt" event={"ID":"daf9bde1-9416-46d4-89ed-e81e287a923f","Type":"ContainerStarted","Data":"baff9e43089bfd6de90a888ba7498866a451539032429996a359421fee20c907"}
Oct 30 22:36:01.230079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:01.229971    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"
Oct 30 22:36:01.230079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:01.229987    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt" event={"ID":"daf9bde1-9416-46d4-89ed-e81e287a923f","Type":"ContainerStarted","Data":"2444259b8e26d05f02c39938695b8d26ccab375ec35123301a343e037d94fd32"}
Oct 30 22:36:01.350558 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:01.350498    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4" podStartSLOduration=1.35047796 podStartE2EDuration="1.35047796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:01.344857591 +0000 UTC m=+3212.079220625" watchObservedRunningTime="2026-10-30 22:36:01.35047796 +0000 UTC m=+3212.084840993"
Oct 30 22:36:01.466005 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:01.465929    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt" podStartSLOduration=2.465908782 podStartE2EDuration="2.465908782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:01.462869796 +0000 UTC m=+3212.197232825" watchObservedRunningTime="2026-10-30 22:36:01.465908782 +0000 UTC m=+3212.200271814"
Oct 30 22:36:12.239793 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:12.239763    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"
Oct 30 22:36:12.240211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:12.239833    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"
Oct 30 22:36:13.194037 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:13.193786    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"]
Oct 30 22:36:13.194340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:13.194128    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls" podUID="5bb81c50-78ea-4df4-9b3c-3525e293d8c0" containerName="istio-proxy" containerID="cri-o://ddda3111ff48aa64f7c974f02683c9b12b92fa6619509eda26de6e78a419e37b" gracePeriod=30
Oct 30 22:36:13.239847 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:13.239804    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"]
Oct 30 22:36:13.240399 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:13.240009    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq" podUID="c75e16cc-0291-4c10-844c-cd8de419f01f" containerName="istio-proxy" containerID="cri-o://0b8f38ecd59811302b114a1ed88dbea027da4070411018af6995fa3dd22331e1" gracePeriod=30
Oct 30 22:36:13.243416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:13.243386    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"]
Oct 30 22:36:13.243624 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:13.243599    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h" podUID="616b564b-7078-4e2d-9111-980e22584acd" containerName="istio-proxy" containerID="cri-o://27b073bd9b353a16a666e886392017c55b3a9709ed33ca84d216573bc662bc75" gracePeriod=30
Oct 30 22:36:13.246987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:13.246963    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"]
Oct 30 22:36:13.247221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:13.247200    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb" podUID="71ab5f78-088f-4354-b901-760d0b228038" containerName="istio-proxy" containerID="cri-o://d5a86e717dfd0f1f47b4dd422bef5f0b3b3f714d3d27e84c51f804520f19ad7c" gracePeriod=30
Oct 30 22:36:18.355879 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.355842    4553 generic.go:356] "Generic (PLEG): container finished" podID="5bb81c50-78ea-4df4-9b3c-3525e293d8c0" containerID="ddda3111ff48aa64f7c974f02683c9b12b92fa6619509eda26de6e78a419e37b" exitCode=0
Oct 30 22:36:18.356385 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.356064    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls" event={"ID":"5bb81c50-78ea-4df4-9b3c-3525e293d8c0","Type":"ContainerDied","Data":"ddda3111ff48aa64f7c974f02683c9b12b92fa6619509eda26de6e78a419e37b"}
Oct 30 22:36:18.540998 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.540971    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:36:18.567049 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.567016    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:36:18.639961 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.639902    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:36:18.643580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.643556    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:36:18.714855 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.714819    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-envoy\") pod \"c75e16cc-0291-4c10-844c-cd8de419f01f\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") "
Oct 30 22:36:18.715114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.714885    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istiod-ca-cert\") pod \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") "
Oct 30 22:36:18.715114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.714933    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-podinfo\") pod \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") "
Oct 30 22:36:18.715114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.714966    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-workload-certs\") pod \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") "
Oct 30 22:36:18.715114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.715004    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-token\" (UniqueName: \"kubernetes.io/projected/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-token\") pod \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") "
Oct 30 22:36:18.715114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.715100    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-workload-certs\") pod \"c75e16cc-0291-4c10-844c-cd8de419f01f\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") "
Oct 30 22:36:18.715572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.715395    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-workload-certs" pod "5bb81c50-78ea-4df4-9b3c-3525e293d8c0" (UID: "5bb81c50-78ea-4df4-9b3c-3525e293d8c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.715572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.715422    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-workload-certs" pod "c75e16cc-0291-4c10-844c-cd8de419f01f" (UID: "c75e16cc-0291-4c10-844c-cd8de419f01f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.715572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.715528    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-envoy\") pod \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") "
Oct 30 22:36:18.715741 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.715581    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-credential-socket\") pod \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") "
Oct 30 22:36:18.715741 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.715638    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75e16cc-0291-4c10-844c-cd8de419f01f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75e16cc-0291-4c10-844c-cd8de419f01f-istiod-ca-cert\") pod \"c75e16cc-0291-4c10-844c-cd8de419f01f\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") "
Oct 30 22:36:18.715741 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.715676    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-token\" (UniqueName: \"kubernetes.io/projected/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-token\") pod \"c75e16cc-0291-4c10-844c-cd8de419f01f\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") "
Oct 30 22:36:18.715741 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.715710    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istiod-ca-cert" pod "5bb81c50-78ea-4df4-9b3c-3525e293d8c0" (UID: "5bb81c50-78ea-4df4-9b3c-3525e293d8c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:18.715964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.715774    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-podinfo\") pod \"c75e16cc-0291-4c10-844c-cd8de419f01f\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") "
Oct 30 22:36:18.715964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.715799    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75e16cc-0291-4c10-844c-cd8de419f01f-kube-api-access-ltbks\" (UniqueName: \"kubernetes.io/projected/c75e16cc-0291-4c10-844c-cd8de419f01f-kube-api-access-ltbks\") pod \"c75e16cc-0291-4c10-844c-cd8de419f01f\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") "
Oct 30 22:36:18.715964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.715846    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-workload-socket\") pod \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") "
Oct 30 22:36:18.715964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.715846    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-credential-socket" pod "5bb81c50-78ea-4df4-9b3c-3525e293d8c0" (UID: "5bb81c50-78ea-4df4-9b3c-3525e293d8c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.715964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.715873    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-workload-socket\") pod \"c75e16cc-0291-4c10-844c-cd8de419f01f\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") "
Oct 30 22:36:18.715964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.715936    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-kube-api-access-rtsk7\" (UniqueName: \"kubernetes.io/projected/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-kube-api-access-rtsk7\") pod \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") "
Oct 30 22:36:18.717439 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.716003    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-data\") pod \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\" (UID: \"5bb81c50-78ea-4df4-9b3c-3525e293d8c0\") "
Oct 30 22:36:18.717439 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.716058    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-credential-socket\") pod \"c75e16cc-0291-4c10-844c-cd8de419f01f\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") "
Oct 30 22:36:18.717439 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.716086    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75e16cc-0291-4c10-844c-cd8de419f01f-istiod-ca-cert" pod "c75e16cc-0291-4c10-844c-cd8de419f01f" (UID: "c75e16cc-0291-4c10-844c-cd8de419f01f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:18.717439 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.716166    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-workload-socket" pod "c75e16cc-0291-4c10-844c-cd8de419f01f" (UID: "c75e16cc-0291-4c10-844c-cd8de419f01f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.717439 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.716365    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-data\") pod \"c75e16cc-0291-4c10-844c-cd8de419f01f\" (UID: \"c75e16cc-0291-4c10-844c-cd8de419f01f\") "
Oct 30 22:36:18.717439 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.717072    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.717439 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.717094    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75e16cc-0291-4c10-844c-cd8de419f01f-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.717439 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.717215    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.717439 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.717235    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.717439 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.717250    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.717439 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.717267    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.718124 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.717458    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-data" pod "c75e16cc-0291-4c10-844c-cd8de419f01f" (UID: "c75e16cc-0291-4c10-844c-cd8de419f01f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.718124 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.717649    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-credential-socket" pod "c75e16cc-0291-4c10-844c-cd8de419f01f" (UID: "c75e16cc-0291-4c10-844c-cd8de419f01f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.718569 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.718541    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-envoy" pod "c75e16cc-0291-4c10-844c-cd8de419f01f" (UID: "c75e16cc-0291-4c10-844c-cd8de419f01f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.718666 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.718619    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-podinfo" pod "5bb81c50-78ea-4df4-9b3c-3525e293d8c0" (UID: "5bb81c50-78ea-4df4-9b3c-3525e293d8c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:18.718666 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.718600    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-podinfo" pod "c75e16cc-0291-4c10-844c-cd8de419f01f" (UID: "c75e16cc-0291-4c10-844c-cd8de419f01f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:18.719060 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.719033    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-workload-socket" pod "5bb81c50-78ea-4df4-9b3c-3525e293d8c0" (UID: "5bb81c50-78ea-4df4-9b3c-3525e293d8c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.719843 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.719806    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-data" pod "5bb81c50-78ea-4df4-9b3c-3525e293d8c0" (UID: "5bb81c50-78ea-4df4-9b3c-3525e293d8c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.720087 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.720054    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-kube-api-access-rtsk7" pod "5bb81c50-78ea-4df4-9b3c-3525e293d8c0" (UID: "5bb81c50-78ea-4df4-9b3c-3525e293d8c0"). InnerVolumeSpecName "kube-api-access-rtsk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:18.720246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.720225    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-token" pod "c75e16cc-0291-4c10-844c-cd8de419f01f" (UID: "c75e16cc-0291-4c10-844c-cd8de419f01f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:18.720513 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.720486    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-envoy" pod "5bb81c50-78ea-4df4-9b3c-3525e293d8c0" (UID: "5bb81c50-78ea-4df4-9b3c-3525e293d8c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.720718 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.720699    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75e16cc-0291-4c10-844c-cd8de419f01f-kube-api-access-ltbks" pod "c75e16cc-0291-4c10-844c-cd8de419f01f" (UID: "c75e16cc-0291-4c10-844c-cd8de419f01f"). InnerVolumeSpecName "kube-api-access-ltbks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:18.720845 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.720832    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-token" pod "5bb81c50-78ea-4df4-9b3c-3525e293d8c0" (UID: "5bb81c50-78ea-4df4-9b3c-3525e293d8c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:18.818019 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.817927    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/616b564b-7078-4e2d-9111-980e22584acd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616b564b-7078-4e2d-9111-980e22584acd-istiod-ca-cert\") pod \"616b564b-7078-4e2d-9111-980e22584acd\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") "
Oct 30 22:36:18.818019 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.817974    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-istio-envoy\") pod \"616b564b-7078-4e2d-9111-980e22584acd\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") "
Oct 30 22:36:18.818314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818025    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-workload-socket\") pod \"71ab5f78-088f-4354-b901-760d0b228038\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") "
Oct 30 22:36:18.818314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818046    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616b564b-7078-4e2d-9111-980e22584acd-istio-token\" (UniqueName: \"kubernetes.io/projected/616b564b-7078-4e2d-9111-980e22584acd-istio-token\") pod \"616b564b-7078-4e2d-9111-980e22584acd\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") "
Oct 30 22:36:18.818314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818062    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-workload-certs\") pod \"616b564b-7078-4e2d-9111-980e22584acd\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") "
Oct 30 22:36:18.818314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818105    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-istio-data\") pod \"616b564b-7078-4e2d-9111-980e22584acd\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") "
Oct 30 22:36:18.818314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818179    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-workload-certs\") pod \"71ab5f78-088f-4354-b901-760d0b228038\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") "
Oct 30 22:36:18.818314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818229    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ab5f78-088f-4354-b901-760d0b228038-istio-token\" (UniqueName: \"kubernetes.io/projected/71ab5f78-088f-4354-b901-760d0b228038-istio-token\") pod \"71ab5f78-088f-4354-b901-760d0b228038\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") "
Oct 30 22:36:18.818314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818284    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-credential-socket\") pod \"616b564b-7078-4e2d-9111-980e22584acd\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") "
Oct 30 22:36:18.818698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818327    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ab5f78-088f-4354-b901-760d0b228038-kube-api-access-2wcln\" (UniqueName: \"kubernetes.io/projected/71ab5f78-088f-4354-b901-760d0b228038-kube-api-access-2wcln\") pod \"71ab5f78-088f-4354-b901-760d0b228038\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") "
Oct 30 22:36:18.818698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818386    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ab5f78-088f-4354-b901-760d0b228038-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ab5f78-088f-4354-b901-760d0b228038-istiod-ca-cert\") pod \"71ab5f78-088f-4354-b901-760d0b228038\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") "
Oct 30 22:36:18.818698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818422    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/616b564b-7078-4e2d-9111-980e22584acd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616b564b-7078-4e2d-9111-980e22584acd-istio-podinfo\") pod \"616b564b-7078-4e2d-9111-980e22584acd\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") "
Oct 30 22:36:18.818698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818453    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-istio-data\") pod \"71ab5f78-088f-4354-b901-760d0b228038\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") "
Oct 30 22:36:18.818698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818481    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-workload-socket\") pod \"616b564b-7078-4e2d-9111-980e22584acd\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") "
Oct 30 22:36:18.818698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818504    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-istio-data" pod "616b564b-7078-4e2d-9111-980e22584acd" (UID: "616b564b-7078-4e2d-9111-980e22584acd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.818698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818523    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616b564b-7078-4e2d-9111-980e22584acd-kube-api-access-tlt6z\" (UniqueName: \"kubernetes.io/projected/616b564b-7078-4e2d-9111-980e22584acd-kube-api-access-tlt6z\") pod \"616b564b-7078-4e2d-9111-980e22584acd\" (UID: \"616b564b-7078-4e2d-9111-980e22584acd\") "
Oct 30 22:36:18.818698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818521    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616b564b-7078-4e2d-9111-980e22584acd-istiod-ca-cert" pod "616b564b-7078-4e2d-9111-980e22584acd" (UID: "616b564b-7078-4e2d-9111-980e22584acd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:18.818698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818511    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-workload-certs" pod "616b564b-7078-4e2d-9111-980e22584acd" (UID: "616b564b-7078-4e2d-9111-980e22584acd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.818698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818529    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-workload-certs" pod "71ab5f78-088f-4354-b901-760d0b228038" (UID: "71ab5f78-088f-4354-b901-760d0b228038"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.818698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818603    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71ab5f78-088f-4354-b901-760d0b228038-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ab5f78-088f-4354-b901-760d0b228038-istio-podinfo\") pod \"71ab5f78-088f-4354-b901-760d0b228038\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") "
Oct 30 22:36:18.818698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818694    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-credential-socket\") pod \"71ab5f78-088f-4354-b901-760d0b228038\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") "
Oct 30 22:36:18.819391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818775    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-credential-socket" pod "616b564b-7078-4e2d-9111-980e22584acd" (UID: "616b564b-7078-4e2d-9111-980e22584acd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.819391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818798    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-istio-data" pod "71ab5f78-088f-4354-b901-760d0b228038" (UID: "71ab5f78-088f-4354-b901-760d0b228038"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.819391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818805    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-istio-envoy\") pod \"71ab5f78-088f-4354-b901-760d0b228038\" (UID: \"71ab5f78-088f-4354-b901-760d0b228038\") "
Oct 30 22:36:18.819391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.818868    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ab5f78-088f-4354-b901-760d0b228038-istiod-ca-cert" pod "71ab5f78-088f-4354-b901-760d0b228038" (UID: "71ab5f78-088f-4354-b901-760d0b228038"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:18.819391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.819207    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-workload-socket" pod "616b564b-7078-4e2d-9111-980e22584acd" (UID: "616b564b-7078-4e2d-9111-980e22584acd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.819391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.819319    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-workload-socket" pod "71ab5f78-088f-4354-b901-760d0b228038" (UID: "71ab5f78-088f-4354-b901-760d0b228038"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.819934 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.819560    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-credential-socket" pod "71ab5f78-088f-4354-b901-760d0b228038" (UID: "71ab5f78-088f-4354-b901-760d0b228038"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820348    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820377    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820391    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820405    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820419    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820432    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820446    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820459    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820472    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820485    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820499    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71ab5f78-088f-4354-b901-760d0b228038-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820511    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820525    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820540    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820554    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltbks\" (UniqueName: \"kubernetes.io/projected/c75e16cc-0291-4c10-844c-cd8de419f01f-kube-api-access-ltbks\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820569    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820583    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820597    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtsk7\" (UniqueName: \"kubernetes.io/projected/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-kube-api-access-rtsk7\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820612    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb81c50-78ea-4df4-9b3c-3525e293d8c0-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820627    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820641    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/616b564b-7078-4e2d-9111-980e22584acd-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.821159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.820656    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75e16cc-0291-4c10-844c-cd8de419f01f-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.822644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.822615    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ab5f78-088f-4354-b901-760d0b228038-kube-api-access-2wcln" pod "71ab5f78-088f-4354-b901-760d0b228038" (UID: "71ab5f78-088f-4354-b901-760d0b228038"). InnerVolumeSpecName "kube-api-access-2wcln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:18.822765 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.822659    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71ab5f78-088f-4354-b901-760d0b228038-istio-podinfo" pod "71ab5f78-088f-4354-b901-760d0b228038" (UID: "71ab5f78-088f-4354-b901-760d0b228038"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:18.822837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.822774    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616b564b-7078-4e2d-9111-980e22584acd-istio-token" pod "616b564b-7078-4e2d-9111-980e22584acd" (UID: "616b564b-7078-4e2d-9111-980e22584acd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:18.822837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.822821    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616b564b-7078-4e2d-9111-980e22584acd-kube-api-access-tlt6z" pod "616b564b-7078-4e2d-9111-980e22584acd" (UID: "616b564b-7078-4e2d-9111-980e22584acd"). InnerVolumeSpecName "kube-api-access-tlt6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:18.822948 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.822851    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ab5f78-088f-4354-b901-760d0b228038-istio-token" pod "71ab5f78-088f-4354-b901-760d0b228038" (UID: "71ab5f78-088f-4354-b901-760d0b228038"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:18.822948 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.822889    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-istio-envoy" pod "71ab5f78-088f-4354-b901-760d0b228038" (UID: "71ab5f78-088f-4354-b901-760d0b228038"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.823031 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.823013    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-istio-envoy" pod "616b564b-7078-4e2d-9111-980e22584acd" (UID: "616b564b-7078-4e2d-9111-980e22584acd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:18.823129 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.823110    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/616b564b-7078-4e2d-9111-980e22584acd-istio-podinfo" pod "616b564b-7078-4e2d-9111-980e22584acd" (UID: "616b564b-7078-4e2d-9111-980e22584acd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:18.922391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.922352    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71ab5f78-088f-4354-b901-760d0b228038-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.922391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.922384    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/616b564b-7078-4e2d-9111-980e22584acd-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.922391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.922396    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/616b564b-7078-4e2d-9111-980e22584acd-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.922657 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.922409    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71ab5f78-088f-4354-b901-760d0b228038-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.922657 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.922422    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wcln\" (UniqueName: \"kubernetes.io/projected/71ab5f78-088f-4354-b901-760d0b228038-kube-api-access-2wcln\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.922657 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.922434    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/616b564b-7078-4e2d-9111-980e22584acd-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.922657 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.922448    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlt6z\" (UniqueName: \"kubernetes.io/projected/616b564b-7078-4e2d-9111-980e22584acd-kube-api-access-tlt6z\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:18.922657 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:18.922459    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71ab5f78-088f-4354-b901-760d0b228038-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:19.363757 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.363721    4553 generic.go:356] "Generic (PLEG): container finished" podID="c75e16cc-0291-4c10-844c-cd8de419f01f" containerID="0b8f38ecd59811302b114a1ed88dbea027da4070411018af6995fa3dd22331e1" exitCode=0
Oct 30 22:36:19.364240 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.363778    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"
Oct 30 22:36:19.364240 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.363813    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq" event={"ID":"c75e16cc-0291-4c10-844c-cd8de419f01f","Type":"ContainerDied","Data":"0b8f38ecd59811302b114a1ed88dbea027da4070411018af6995fa3dd22331e1"}
Oct 30 22:36:19.364240 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.363877    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq" event={"ID":"c75e16cc-0291-4c10-844c-cd8de419f01f","Type":"ContainerDied","Data":"50519ca02b10ef7e60fd1744240afb3b5a33fa7bb91fc9917cc2ee76a38b87d8"}
Oct 30 22:36:19.364240 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.363899    4553 scope.go:122] "RemoveContainer" containerID="0b8f38ecd59811302b114a1ed88dbea027da4070411018af6995fa3dd22331e1"
Oct 30 22:36:19.365542 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.365514    4553 generic.go:356] "Generic (PLEG): container finished" podID="616b564b-7078-4e2d-9111-980e22584acd" containerID="27b073bd9b353a16a666e886392017c55b3a9709ed33ca84d216573bc662bc75" exitCode=0
Oct 30 22:36:19.365617 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.365568    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"
Oct 30 22:36:19.365681 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.365645    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h" event={"ID":"616b564b-7078-4e2d-9111-980e22584acd","Type":"ContainerDied","Data":"27b073bd9b353a16a666e886392017c55b3a9709ed33ca84d216573bc662bc75"}
Oct 30 22:36:19.365726 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.365695    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h" event={"ID":"616b564b-7078-4e2d-9111-980e22584acd","Type":"ContainerDied","Data":"d48af2281888c617905362a225767860ec96828b60d882a443ba778c7ae10b2c"}
Oct 30 22:36:19.367555 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.367527    4553 generic.go:356] "Generic (PLEG): container finished" podID="71ab5f78-088f-4354-b901-760d0b228038" containerID="d5a86e717dfd0f1f47b4dd422bef5f0b3b3f714d3d27e84c51f804520f19ad7c" exitCode=0
Oct 30 22:36:19.367680 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.367578    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"
Oct 30 22:36:19.367680 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.367606    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb" event={"ID":"71ab5f78-088f-4354-b901-760d0b228038","Type":"ContainerDied","Data":"d5a86e717dfd0f1f47b4dd422bef5f0b3b3f714d3d27e84c51f804520f19ad7c"}
Oct 30 22:36:19.367680 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.367646    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb" event={"ID":"71ab5f78-088f-4354-b901-760d0b228038","Type":"ContainerDied","Data":"d499238aeff8c609dfd98c5372a8eb3afbb70b65b60a725cafe99a16b60f5db1"}
Oct 30 22:36:19.369320 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.369293    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls" event={"ID":"5bb81c50-78ea-4df4-9b3c-3525e293d8c0","Type":"ContainerDied","Data":"8594b19746d67ff9afa2720df3f8adb1677616874b3956b32d1aaab66fbe9924"}
Oct 30 22:36:19.369320 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.369324    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"
Oct 30 22:36:19.378014 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.377789    4553 scope.go:122] "RemoveContainer" containerID="0b8f38ecd59811302b114a1ed88dbea027da4070411018af6995fa3dd22331e1"
Oct 30 22:36:19.378195 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:36:19.378098    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8f38ecd59811302b114a1ed88dbea027da4070411018af6995fa3dd22331e1\": container with ID starting with 0b8f38ecd59811302b114a1ed88dbea027da4070411018af6995fa3dd22331e1 not found: ID does not exist" containerID="0b8f38ecd59811302b114a1ed88dbea027da4070411018af6995fa3dd22331e1"
Oct 30 22:36:19.378195 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.378121    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8f38ecd59811302b114a1ed88dbea027da4070411018af6995fa3dd22331e1"} err="failed to get container status \"0b8f38ecd59811302b114a1ed88dbea027da4070411018af6995fa3dd22331e1\": rpc error: code = NotFound desc = could not find container \"0b8f38ecd59811302b114a1ed88dbea027da4070411018af6995fa3dd22331e1\": container with ID starting with 0b8f38ecd59811302b114a1ed88dbea027da4070411018af6995fa3dd22331e1 not found: ID does not exist"
Oct 30 22:36:19.378195 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.378161    4553 scope.go:122] "RemoveContainer" containerID="27b073bd9b353a16a666e886392017c55b3a9709ed33ca84d216573bc662bc75"
Oct 30 22:36:19.389717 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.389693    4553 scope.go:122] "RemoveContainer" containerID="27b073bd9b353a16a666e886392017c55b3a9709ed33ca84d216573bc662bc75"
Oct 30 22:36:19.389963 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:36:19.389942    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b073bd9b353a16a666e886392017c55b3a9709ed33ca84d216573bc662bc75\": container with ID starting with 27b073bd9b353a16a666e886392017c55b3a9709ed33ca84d216573bc662bc75 not found: ID does not exist" containerID="27b073bd9b353a16a666e886392017c55b3a9709ed33ca84d216573bc662bc75"
Oct 30 22:36:19.390015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.389972    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b073bd9b353a16a666e886392017c55b3a9709ed33ca84d216573bc662bc75"} err="failed to get container status \"27b073bd9b353a16a666e886392017c55b3a9709ed33ca84d216573bc662bc75\": rpc error: code = NotFound desc = could not find container \"27b073bd9b353a16a666e886392017c55b3a9709ed33ca84d216573bc662bc75\": container with ID starting with 27b073bd9b353a16a666e886392017c55b3a9709ed33ca84d216573bc662bc75 not found: ID does not exist"
Oct 30 22:36:19.390015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.389991    4553 scope.go:122] "RemoveContainer" containerID="d5a86e717dfd0f1f47b4dd422bef5f0b3b3f714d3d27e84c51f804520f19ad7c"
Oct 30 22:36:19.400145 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.400113    4553 scope.go:122] "RemoveContainer" containerID="d5a86e717dfd0f1f47b4dd422bef5f0b3b3f714d3d27e84c51f804520f19ad7c"
Oct 30 22:36:19.400451 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:36:19.400432    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5a86e717dfd0f1f47b4dd422bef5f0b3b3f714d3d27e84c51f804520f19ad7c\": container with ID starting with d5a86e717dfd0f1f47b4dd422bef5f0b3b3f714d3d27e84c51f804520f19ad7c not found: ID does not exist" containerID="d5a86e717dfd0f1f47b4dd422bef5f0b3b3f714d3d27e84c51f804520f19ad7c"
Oct 30 22:36:19.400525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.400461    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5a86e717dfd0f1f47b4dd422bef5f0b3b3f714d3d27e84c51f804520f19ad7c"} err="failed to get container status \"d5a86e717dfd0f1f47b4dd422bef5f0b3b3f714d3d27e84c51f804520f19ad7c\": rpc error: code = NotFound desc = could not find container \"d5a86e717dfd0f1f47b4dd422bef5f0b3b3f714d3d27e84c51f804520f19ad7c\": container with ID starting with d5a86e717dfd0f1f47b4dd422bef5f0b3b3f714d3d27e84c51f804520f19ad7c not found: ID does not exist"
Oct 30 22:36:19.400525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.400479    4553 scope.go:122] "RemoveContainer" containerID="ddda3111ff48aa64f7c974f02683c9b12b92fa6619509eda26de6e78a419e37b"
Oct 30 22:36:19.535222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.535121    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"]
Oct 30 22:36:19.585361 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.585328    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-2mq4h"]
Oct 30 22:36:19.650483 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.650443    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"]
Oct 30 22:36:19.653831 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.653805    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-t8wsb"]
Oct 30 22:36:19.764013 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.763978    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"]
Oct 30 22:36:19.769500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.769472    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-nfwzq"]
Oct 30 22:36:19.880929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.880895    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"]
Oct 30 22:36:19.884769 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:19.884743    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90e0d5-openshift-default-6cbd58cf8c-lh9ls"]
Oct 30 22:36:21.393777 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:21.393745    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb81c50-78ea-4df4-9b3c-3525e293d8c0" path="/var/lib/kubelet/pods/5bb81c50-78ea-4df4-9b3c-3525e293d8c0/volumes"
Oct 30 22:36:21.394281 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:21.394266    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="616b564b-7078-4e2d-9111-980e22584acd" path="/var/lib/kubelet/pods/616b564b-7078-4e2d-9111-980e22584acd/volumes"
Oct 30 22:36:21.394698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:21.394686    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ab5f78-088f-4354-b901-760d0b228038" path="/var/lib/kubelet/pods/71ab5f78-088f-4354-b901-760d0b228038/volumes"
Oct 30 22:36:21.395486 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:21.395471    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75e16cc-0291-4c10-844c-cd8de419f01f" path="/var/lib/kubelet/pods/c75e16cc-0291-4c10-844c-cd8de419f01f/volumes"
Oct 30 22:36:28.202776 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:36:28.202743    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:36:37.496604 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:37.496566    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"]
Oct 30 22:36:37.497225 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:37.496860    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9" podUID="607d78fc-1484-4719-beb9-e6f32376729d" containerName="istio-proxy" containerID="cri-o://c9f3c1ebcf3d550e7f9781d147f952c03b5849e7775976447d1f9d7bcaeb0bdf" gracePeriod=30
Oct 30 22:36:42.778105 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.778038    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:36:42.943401 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.943362    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/607d78fc-1484-4719-beb9-e6f32376729d-istio-token\" (UniqueName: \"kubernetes.io/projected/607d78fc-1484-4719-beb9-e6f32376729d-istio-token\") pod \"607d78fc-1484-4719-beb9-e6f32376729d\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") "
Oct 30 22:36:42.943401 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.943410    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-credential-socket\") pod \"607d78fc-1484-4719-beb9-e6f32376729d\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") "
Oct 30 22:36:42.943652 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.943486    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-workload-socket\") pod \"607d78fc-1484-4719-beb9-e6f32376729d\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") "
Oct 30 22:36:42.943652 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.943531    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/607d78fc-1484-4719-beb9-e6f32376729d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/607d78fc-1484-4719-beb9-e6f32376729d-istio-podinfo\") pod \"607d78fc-1484-4719-beb9-e6f32376729d\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") "
Oct 30 22:36:42.943652 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.943614    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-istio-envoy\") pod \"607d78fc-1484-4719-beb9-e6f32376729d\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") "
Oct 30 22:36:42.943834 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.943726    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/607d78fc-1484-4719-beb9-e6f32376729d-kube-api-access-lnzkm\" (UniqueName: \"kubernetes.io/projected/607d78fc-1484-4719-beb9-e6f32376729d-kube-api-access-lnzkm\") pod \"607d78fc-1484-4719-beb9-e6f32376729d\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") "
Oct 30 22:36:42.943834 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.943770    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-credential-socket" pod "607d78fc-1484-4719-beb9-e6f32376729d" (UID: "607d78fc-1484-4719-beb9-e6f32376729d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:42.943834 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.943797    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/607d78fc-1484-4719-beb9-e6f32376729d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/607d78fc-1484-4719-beb9-e6f32376729d-istiod-ca-cert\") pod \"607d78fc-1484-4719-beb9-e6f32376729d\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") "
Oct 30 22:36:42.943834 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.943818    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-workload-socket" pod "607d78fc-1484-4719-beb9-e6f32376729d" (UID: "607d78fc-1484-4719-beb9-e6f32376729d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:42.944046 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.943856    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-istio-data\") pod \"607d78fc-1484-4719-beb9-e6f32376729d\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") "
Oct 30 22:36:42.944163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.944114    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-istio-data" pod "607d78fc-1484-4719-beb9-e6f32376729d" (UID: "607d78fc-1484-4719-beb9-e6f32376729d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:42.944239 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.944216    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-workload-certs\") pod \"607d78fc-1484-4719-beb9-e6f32376729d\" (UID: \"607d78fc-1484-4719-beb9-e6f32376729d\") "
Oct 30 22:36:42.944368 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.944341    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/607d78fc-1484-4719-beb9-e6f32376729d-istiod-ca-cert" pod "607d78fc-1484-4719-beb9-e6f32376729d" (UID: "607d78fc-1484-4719-beb9-e6f32376729d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:42.944637 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.944451    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-workload-certs" pod "607d78fc-1484-4719-beb9-e6f32376729d" (UID: "607d78fc-1484-4719-beb9-e6f32376729d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:42.944849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.944827    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:42.944946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.944848    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:42.944946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.944864    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:42.944946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.944877    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:42.944946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.944891    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/607d78fc-1484-4719-beb9-e6f32376729d-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:42.945979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.945957    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/607d78fc-1484-4719-beb9-e6f32376729d-istio-token" pod "607d78fc-1484-4719-beb9-e6f32376729d" (UID: "607d78fc-1484-4719-beb9-e6f32376729d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:42.946513 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.946494    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/607d78fc-1484-4719-beb9-e6f32376729d-istio-podinfo" pod "607d78fc-1484-4719-beb9-e6f32376729d" (UID: "607d78fc-1484-4719-beb9-e6f32376729d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:42.946605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.946506    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-istio-envoy" pod "607d78fc-1484-4719-beb9-e6f32376729d" (UID: "607d78fc-1484-4719-beb9-e6f32376729d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:42.946672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:42.946643    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/607d78fc-1484-4719-beb9-e6f32376729d-kube-api-access-lnzkm" pod "607d78fc-1484-4719-beb9-e6f32376729d" (UID: "607d78fc-1484-4719-beb9-e6f32376729d"). InnerVolumeSpecName "kube-api-access-lnzkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:43.046203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:43.046107    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/607d78fc-1484-4719-beb9-e6f32376729d-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:43.046203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:43.046170    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/607d78fc-1484-4719-beb9-e6f32376729d-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:43.046203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:43.046181    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/607d78fc-1484-4719-beb9-e6f32376729d-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:43.046203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:43.046190    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnzkm\" (UniqueName: \"kubernetes.io/projected/607d78fc-1484-4719-beb9-e6f32376729d-kube-api-access-lnzkm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:43.531847 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:43.531808    4553 generic.go:356] "Generic (PLEG): container finished" podID="607d78fc-1484-4719-beb9-e6f32376729d" containerID="c9f3c1ebcf3d550e7f9781d147f952c03b5849e7775976447d1f9d7bcaeb0bdf" exitCode=0
Oct 30 22:36:43.532021 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:43.531879    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9" event={"ID":"607d78fc-1484-4719-beb9-e6f32376729d","Type":"ContainerDied","Data":"c9f3c1ebcf3d550e7f9781d147f952c03b5849e7775976447d1f9d7bcaeb0bdf"}
Oct 30 22:36:43.532021 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:43.531887    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"
Oct 30 22:36:43.532021 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:43.531933    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9" event={"ID":"607d78fc-1484-4719-beb9-e6f32376729d","Type":"ContainerDied","Data":"1e26d893ae18b5e260b0f833c3a2fe5e924b7e54fc865fdfc52e01508cba0eac"}
Oct 30 22:36:43.532021 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:43.531962    4553 scope.go:122] "RemoveContainer" containerID="c9f3c1ebcf3d550e7f9781d147f952c03b5849e7775976447d1f9d7bcaeb0bdf"
Oct 30 22:36:43.542777 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:43.542756    4553 scope.go:122] "RemoveContainer" containerID="c9f3c1ebcf3d550e7f9781d147f952c03b5849e7775976447d1f9d7bcaeb0bdf"
Oct 30 22:36:43.543053 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:36:43.543033    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f3c1ebcf3d550e7f9781d147f952c03b5849e7775976447d1f9d7bcaeb0bdf\": container with ID starting with c9f3c1ebcf3d550e7f9781d147f952c03b5849e7775976447d1f9d7bcaeb0bdf not found: ID does not exist" containerID="c9f3c1ebcf3d550e7f9781d147f952c03b5849e7775976447d1f9d7bcaeb0bdf"
Oct 30 22:36:43.543168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:43.543062    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f3c1ebcf3d550e7f9781d147f952c03b5849e7775976447d1f9d7bcaeb0bdf"} err="failed to get container status \"c9f3c1ebcf3d550e7f9781d147f952c03b5849e7775976447d1f9d7bcaeb0bdf\": rpc error: code = NotFound desc = could not find container \"c9f3c1ebcf3d550e7f9781d147f952c03b5849e7775976447d1f9d7bcaeb0bdf\": container with ID starting with c9f3c1ebcf3d550e7f9781d147f952c03b5849e7775976447d1f9d7bcaeb0bdf not found: ID does not exist"
Oct 30 22:36:43.700750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:43.700657    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"]
Oct 30 22:36:43.704528 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:43.704506    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-cv6k9"]
Oct 30 22:36:45.394387 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:45.394352    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="607d78fc-1484-4719-beb9-e6f32376729d" path="/var/lib/kubelet/pods/607d78fc-1484-4719-beb9-e6f32376729d/volumes"
Oct 30 22:36:57.986282 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.986247    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"]
Oct 30 22:36:57.986931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.986917    4553 cpu_manager.go:414] "containerMap: removing container" podUID="616b564b-7078-4e2d-9111-980e22584acd" containerName="istio-proxy"
Oct 30 22:36:57.986978 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.986932    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616b564b-7078-4e2d-9111-980e22584acd" containerName="istio-proxy"
Oct 30 22:36:57.986978 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.986944    4553 cpu_manager.go:414] "containerMap: removing container" podUID="c75e16cc-0291-4c10-844c-cd8de419f01f" containerName="istio-proxy"
Oct 30 22:36:57.986978 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.986950    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75e16cc-0291-4c10-844c-cd8de419f01f" containerName="istio-proxy"
Oct 30 22:36:57.986978 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.986961    4553 cpu_manager.go:414] "containerMap: removing container" podUID="607d78fc-1484-4719-beb9-e6f32376729d" containerName="istio-proxy"
Oct 30 22:36:57.986978 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.986966    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="607d78fc-1484-4719-beb9-e6f32376729d" containerName="istio-proxy"
Oct 30 22:36:57.986978 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.986974    4553 cpu_manager.go:414] "containerMap: removing container" podUID="71ab5f78-088f-4354-b901-760d0b228038" containerName="istio-proxy"
Oct 30 22:36:57.987198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.986981    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ab5f78-088f-4354-b901-760d0b228038" containerName="istio-proxy"
Oct 30 22:36:57.987198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.987011    4553 cpu_manager.go:414] "containerMap: removing container" podUID="5bb81c50-78ea-4df4-9b3c-3525e293d8c0" containerName="istio-proxy"
Oct 30 22:36:57.987198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.987015    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb81c50-78ea-4df4-9b3c-3525e293d8c0" containerName="istio-proxy"
Oct 30 22:36:57.987198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.987098    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ab5f78-088f-4354-b901-760d0b228038" containerName="istio-proxy"
Oct 30 22:36:57.987198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.987113    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb81c50-78ea-4df4-9b3c-3525e293d8c0" containerName="istio-proxy"
Oct 30 22:36:57.987198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.987122    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="616b564b-7078-4e2d-9111-980e22584acd" containerName="istio-proxy"
Oct 30 22:36:57.987198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.987152    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75e16cc-0291-4c10-844c-cd8de419f01f" containerName="istio-proxy"
Oct 30 22:36:57.987198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.987164    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="607d78fc-1484-4719-beb9-e6f32376729d" containerName="istio-proxy"
Oct 30 22:36:57.992005 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:57.991985    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.047625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.047589    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"]
Oct 30 22:36:58.093624 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.093587    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-b4deef\"/\"auth-test-model-b4deef-kserve-self-signed-certs\""
Oct 30 22:36:58.093624 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.093597    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-b4deef\"/\"kube-root-ca.crt\""
Oct 30 22:36:58.096946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.096926    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-b4deef\"/\"openshift-service-ca.crt\""
Oct 30 22:36:58.097013 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.096926    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-b4deef\"/\"default-dockercfg-4bc7t\""
Oct 30 22:36:58.136632 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.136261    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-model-cache\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.136632 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.136395    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-dshm\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.136632 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.136457    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-home\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.136632 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.136500    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b996db18-28c1-47da-889e-b99c118f1c42-tls-certs\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.136632 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.136552    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-tmp-dir\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.136632 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.136570    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdf5m\" (UniqueName: \"kubernetes.io/projected/b996db18-28c1-47da-889e-b99c118f1c42-kube-api-access-gdf5m\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.204625 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:36:58.204592    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:36:58.238017 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.237913    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-dshm\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.238017 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.237995    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-home\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.238304 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.238052    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b996db18-28c1-47da-889e-b99c118f1c42-tls-certs\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.238304 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.238111    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-tmp-dir\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.238304 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.238181    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdf5m\" (UniqueName: \"kubernetes.io/projected/b996db18-28c1-47da-889e-b99c118f1c42-kube-api-access-gdf5m\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.238304 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.238267    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-model-cache\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.238523 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.238455    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-home\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.238523 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.238504    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-tmp-dir\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.238624 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.238559    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-model-cache\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.240551 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.240531    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-dshm\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.240827 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.240812    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b996db18-28c1-47da-889e-b99c118f1c42-tls-certs\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.296585 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.296545    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdf5m\" (UniqueName: \"kubernetes.io/projected/b996db18-28c1-47da-889e-b99c118f1c42-kube-api-access-gdf5m\") pod \"auth-test-model-b4deef-kserve-777854dc7-wpmlw\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") " pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.304699 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.304672    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:36:58.685321 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:58.685294    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"]
Oct 30 22:36:58.685922 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:36:58.685895    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb996db18_28c1_47da_889e_b99c118f1c42.slice/crio-fa81857d1c163cfff810925ce7356ef4577e6b19eb99bec512317115b5a914d5 WatchSource:0}: Error finding container fa81857d1c163cfff810925ce7356ef4577e6b19eb99bec512317115b5a914d5: Status 404 returned error can't find the container with id fa81857d1c163cfff810925ce7356ef4577e6b19eb99bec512317115b5a914d5
Oct 30 22:36:59.646784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:59.646741    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw" event={"ID":"b996db18-28c1-47da-889e-b99c118f1c42","Type":"ContainerStarted","Data":"941ee63af6ba3f69b74ace21752fadb2d452dbf0cf95dec5f5ebc7c1686f20ce"}
Oct 30 22:36:59.646784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:59.646799    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw" event={"ID":"b996db18-28c1-47da-889e-b99c118f1c42","Type":"ContainerStarted","Data":"fa81857d1c163cfff810925ce7356ef4577e6b19eb99bec512317115b5a914d5"}
Oct 30 22:36:59.763537 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:59.763479    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw" podStartSLOduration=2.763465037 podStartE2EDuration="2.763465037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:59.763338284 +0000 UTC m=+3270.497701316" watchObservedRunningTime="2026-10-30 22:36:59.763465037 +0000 UTC m=+3270.497828159"
Oct 30 22:36:59.943551 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:59.943465    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"]
Oct 30 22:36:59.947821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:36:59.947798    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:00.002421 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.002360    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-0e450a-serving-cert\""
Oct 30 22:37:00.007024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.006987    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"]
Oct 30 22:37:00.063771 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.063731    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:00.063945 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.063924    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963f2df0-8b48-4caf-8485-1a8dfe58ee70-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:00.064007 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.063998    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5wjd\" (UniqueName: \"kubernetes.io/projected/963f2df0-8b48-4caf-8485-1a8dfe58ee70-kube-api-access-x5wjd\") pod \"maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:00.064091 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.064075    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:00.165541 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.165502    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:00.165765 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.165605    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:00.165765 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:37:00.165675    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-serving-cert: secret "maas-api-e2e-shared-b-w3-0e450a-serving-cert" not found
Oct 30 22:37:00.165765 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:37:00.165751    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-maas-api-tls podName:963f2df0-8b48-4caf-8485-1a8dfe58ee70 nodeName:}" failed. No retries permitted until 2026-10-30 22:37:00.665726435 +0000 UTC m=+3271.400089451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-maas-api-tls") pod "maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt" (UID: "963f2df0-8b48-4caf-8485-1a8dfe58ee70") : secret "maas-api-e2e-shared-b-w3-0e450a-serving-cert" not found
Oct 30 22:37:00.165902 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.165841    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963f2df0-8b48-4caf-8485-1a8dfe58ee70-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:00.165942 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.165918    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5wjd\" (UniqueName: \"kubernetes.io/projected/963f2df0-8b48-4caf-8485-1a8dfe58ee70-kube-api-access-x5wjd\") pod \"maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:00.166797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.166711    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963f2df0-8b48-4caf-8485-1a8dfe58ee70-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:00.169421 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.169396    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:00.242667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.242577    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5wjd\" (UniqueName: \"kubernetes.io/projected/963f2df0-8b48-4caf-8485-1a8dfe58ee70-kube-api-access-x5wjd\") pod \"maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:00.305583 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.305541    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:37:00.672109 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.672073    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:00.674754 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.674736    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:00.862736 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:00.862710    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:01.279879 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:01.279852    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"]
Oct 30 22:37:01.281129 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:37:01.281104    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963f2df0_8b48_4caf_8485_1a8dfe58ee70.slice/crio-4721873145d75cebf080cc37a602fe3a26535545911002dac96053d076722108 WatchSource:0}: Error finding container 4721873145d75cebf080cc37a602fe3a26535545911002dac96053d076722108: Status 404 returned error can't find the container with id 4721873145d75cebf080cc37a602fe3a26535545911002dac96053d076722108
Oct 30 22:37:01.665779 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:01.665729    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt" event={"ID":"963f2df0-8b48-4caf-8485-1a8dfe58ee70","Type":"ContainerStarted","Data":"4721873145d75cebf080cc37a602fe3a26535545911002dac96053d076722108"}
Oct 30 22:37:02.674034 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:02.673997    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt" event={"ID":"963f2df0-8b48-4caf-8485-1a8dfe58ee70","Type":"ContainerStarted","Data":"5fe39366e8edb51ad8a2c3fe333e09f19c69cadf46f61183da194da851b8915e"}
Oct 30 22:37:02.674579 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:02.674076    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:02.794646 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:02.794588    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt" podStartSLOduration=2.817118646 podStartE2EDuration="3.794569939s" totalImagesPullingTime="977.451293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:02.792299916 +0000 UTC m=+3273.526662949" watchObservedRunningTime="2026-10-30 22:37:02.794569939 +0000 UTC m=+3273.528932970"
Oct 30 22:37:08.305278 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:08.305234    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:37:08.318763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:08.318732    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:37:08.687104 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:08.687073    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:37:08.728659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:08.728629    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:37:09.722567 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:09.722527    4553 generic.go:356] "Generic (PLEG): container finished" podID="aff22a9f-32c6-4373-8d9d-62f6ce177851" containerID="02144e37d2ad8dce8951e78a398c810f112810d14266d2e9dbc15cb2bccfad8f" exitCode=0
Oct 30 22:37:09.723029 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:09.722603    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv" event={"ID":"aff22a9f-32c6-4373-8d9d-62f6ce177851","Type":"ContainerDied","Data":"02144e37d2ad8dce8951e78a398c810f112810d14266d2e9dbc15cb2bccfad8f"}
Oct 30 22:37:09.723442 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:09.723423    4553 scope.go:122] "RemoveContainer" containerID="02144e37d2ad8dce8951e78a398c810f112810d14266d2e9dbc15cb2bccfad8f"
Oct 30 22:37:09.725487 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:09.725456    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:37:09.725603 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:09.725531    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:37:09.725603 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:09.725563    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:37:10.740112 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:10.740074    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv" event={"ID":"aff22a9f-32c6-4373-8d9d-62f6ce177851","Type":"ContainerStarted","Data":"ec80e0f5661e87c1cc769fc7604e6904ecbe360165c2579a26660ca992ab223b"}
Oct 30 22:37:11.059338 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:11.059245    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:37:11.059525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:11.059482    4553 patch_prober.go:29] interesting pod/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:37:11.059609 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:11.059539    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv" podUID="aff22a9f-32c6-4373-8d9d-62f6ce177851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 22:37:12.059362 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:12.059331    4553 patch_prober.go:29] interesting pod/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:37:12.059813 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:12.059385    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv" podUID="aff22a9f-32c6-4373-8d9d-62f6ce177851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 22:37:13.059695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:13.059659    4553 patch_prober.go:29] interesting pod/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:37:13.060114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:13.059715    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv" podUID="aff22a9f-32c6-4373-8d9d-62f6ce177851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 22:37:14.059767 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:14.059733    4553 patch_prober.go:29] interesting pod/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:37:14.060232 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:14.059789    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv" podUID="aff22a9f-32c6-4373-8d9d-62f6ce177851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 22:37:15.062741 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:15.062706    4553 patch_prober.go:29] interesting pod/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:37:15.063331 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:15.063299    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv" podUID="aff22a9f-32c6-4373-8d9d-62f6ce177851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 22:37:16.059681 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:16.059650    4553 patch_prober.go:29] interesting pod/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:37:16.059934 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:16.059912    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv" podUID="aff22a9f-32c6-4373-8d9d-62f6ce177851" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Oct 30 22:37:17.064010 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:17.063976    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:37:17.064531 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:17.064375    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:37:17.064945 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:17.064925    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:37:28.205370 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:37:28.205328    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:37:29.960424 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:29.960390    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:37:29.966531 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:29.966507    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:37:52.363950 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.363911    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"]
Oct 30 22:37:52.369721 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.369695    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.425350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.425308    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"]
Oct 30 22:37:52.470937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.470904    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-b1eb3f\"/\"default-dockercfg-nljmj\""
Oct 30 22:37:52.470937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.470904    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-b1eb3f\"/\"openshift-service-ca.crt\""
Oct 30 22:37:52.472659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.472627    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-b1eb3f\"/\"kube-root-ca.crt\""
Oct 30 22:37:52.472659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.472645    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-b1eb3f\"/\"auth-test-model-b1eb3f-kserve-self-signed-certs\""
Oct 30 22:37:52.494165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.494113    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-tls-certs\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.494301 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.494255    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-home\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.494301 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.494283    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgq9s\" (UniqueName: \"kubernetes.io/projected/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-kube-api-access-zgq9s\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.494408 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.494389    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-model-cache\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.494501 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.494487    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-tmp-dir\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.494601 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.494587    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-dshm\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.596513 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.596467    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-home\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.596513 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.596514    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgq9s\" (UniqueName: \"kubernetes.io/projected/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-kube-api-access-zgq9s\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.596814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.596559    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-model-cache\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.596814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.596715    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-tmp-dir\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.596943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.596827    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-dshm\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.596943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.596923    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-home\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.596943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.596929    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-tls-certs\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.597088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.596951    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-model-cache\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.597088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.597009    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-tmp-dir\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.599297 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.599271    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-dshm\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.599578 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.599558    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-tls-certs\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.656035 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.655957    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgq9s\" (UniqueName: \"kubernetes.io/projected/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-kube-api-access-zgq9s\") pod \"auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") " pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:52.683476 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:52.683444    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:53.079187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:53.079157    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"]
Oct 30 22:37:53.080644 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:37:53.080613    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ca88c62_838d_4aa1_8c6f_dc1d639eaad2.slice/crio-07a4e869660a8332925088a484469afd406e97b2e38521004de593757e42aa0d WatchSource:0}: Error finding container 07a4e869660a8332925088a484469afd406e97b2e38521004de593757e42aa0d: Status 404 returned error can't find the container with id 07a4e869660a8332925088a484469afd406e97b2e38521004de593757e42aa0d
Oct 30 22:37:54.036738 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:54.036699    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86" event={"ID":"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2","Type":"ContainerStarted","Data":"7a264d367f7fa196d7b63a5fd4049246562503c56b172dffc0c6a23d23f67a4e"}
Oct 30 22:37:54.037194 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:54.036753    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86" event={"ID":"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2","Type":"ContainerStarted","Data":"07a4e869660a8332925088a484469afd406e97b2e38521004de593757e42aa0d"}
Oct 30 22:37:54.152771 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:54.152712    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86" podStartSLOduration=2.152697368 podStartE2EDuration="2.152697368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:54.152127551 +0000 UTC m=+3324.886490582" watchObservedRunningTime="2026-10-30 22:37:54.152697368 +0000 UTC m=+3324.887060398"
Oct 30 22:37:54.683812 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:37:54.683781    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:37:58.203064 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:37:58.203022    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:38:02.683981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:02.683946    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:38:02.698495 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:02.698463    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:38:03.107895 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:03.107861    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:38:10.628072 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:10.628031    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"]
Oct 30 22:38:10.628614 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:10.628340    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x" podUID="f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" containerName="istio-proxy" containerID="cri-o://40dc8616ea9fdfb5c82376fe70b5f386111fd57317e5a35fa7875f3b37787045" gracePeriod=30
Oct 30 22:38:11.338863 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.338825    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"]
Oct 30 22:38:11.351933 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.351888    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.404072 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.404018    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"]
Oct 30 22:38:11.404771 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.404747    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-9651e9\"/\"kube-root-ca.crt\""
Oct 30 22:38:11.404984 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.404747    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-9651e9\"/\"openshift-service-ca.crt\""
Oct 30 22:38:11.445797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.445767    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-9651e9\"/\"default-dockercfg-l7mrh\""
Oct 30 22:38:11.445988 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.445867    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-9651e9\"/\"explicit-model-9651e9-kserve-self-signed-certs\""
Oct 30 22:38:11.455930 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.455904    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-tmp-dir\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.456046 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.456029    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-home\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.456168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.456120    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-model-cache\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.456278 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.456263    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t6w4\" (UniqueName: \"kubernetes.io/projected/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-kube-api-access-9t6w4\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.456325 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.456313    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-tls-certs\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.456411 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.456400    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-dshm\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.558198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.558165    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-home\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.558393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.558221    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-model-cache\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.558393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.558286    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t6w4\" (UniqueName: \"kubernetes.io/projected/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-kube-api-access-9t6w4\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.558393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.558330    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-tls-certs\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.558393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.558359    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-dshm\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.558676 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.558399    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-tmp-dir\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.558758 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.558677    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-model-cache\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.558758 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.558694    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-home\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.558858 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.558767    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-tmp-dir\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.560966 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.560929    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-dshm\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.561215 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.561197    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-tls-certs\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.617275 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.617193    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t6w4\" (UniqueName: \"kubernetes.io/projected/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-kube-api-access-9t6w4\") pod \"explicit-model-9651e9-kserve-7dcc79f48f-qs98d\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") " pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.667387 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.667350    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:11.773068 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.772756    4553 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-bxr9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:38:11.773068 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:11.772813    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x" podUID="f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:38:12.051364 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:12.051332    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"]
Oct 30 22:38:12.051891 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:38:12.051866    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode12162f0_3b43_4bcf_bd0a_c4298b61dbd9.slice/crio-712b97ea66a133a306d851dc60753f3ed49cd187e5501f243635d5f4b6a9f34c WatchSource:0}: Error finding container 712b97ea66a133a306d851dc60753f3ed49cd187e5501f243635d5f4b6a9f34c: Status 404 returned error can't find the container with id 712b97ea66a133a306d851dc60753f3ed49cd187e5501f243635d5f4b6a9f34c
Oct 30 22:38:12.157333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:12.157295    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d" event={"ID":"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9","Type":"ContainerStarted","Data":"be9bfbc3f1cd3b5161db603626d852956c5ffb904926afccecf6222d81ac4368"}
Oct 30 22:38:12.157480 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:12.157342    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d" event={"ID":"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9","Type":"ContainerStarted","Data":"712b97ea66a133a306d851dc60753f3ed49cd187e5501f243635d5f4b6a9f34c"}
Oct 30 22:38:12.273369 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:12.273319    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d" podStartSLOduration=1.2733046639999999 podStartE2EDuration="1.273304664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:12.272407653 +0000 UTC m=+3343.006770687" watchObservedRunningTime="2026-10-30 22:38:12.273304664 +0000 UTC m=+3343.007667694"
Oct 30 22:38:12.668107 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:12.668073    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:15.904646 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:15.904575    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:38:16.018795 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.018761    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-workload-certs\") pod \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") "
Oct 30 22:38:16.018983 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.018812    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-token\" (UniqueName: \"kubernetes.io/projected/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-token\") pod \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") "
Oct 30 22:38:16.018983 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.018838    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-envoy\") pod \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") "
Oct 30 22:38:16.018983 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.018900    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-data\") pod \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") "
Oct 30 22:38:16.019119 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.019046    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-credential-socket\") pod \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") "
Oct 30 22:38:16.019119 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.019085    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istiod-ca-cert\") pod \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") "
Oct 30 22:38:16.019119 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.019107    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-podinfo\") pod \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") "
Oct 30 22:38:16.019343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.019115    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-workload-certs" pod "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" (UID: "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:16.019343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.019217    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-workload-socket\") pod \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") "
Oct 30 22:38:16.019343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.019271    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-data" pod "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" (UID: "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:16.019517 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.019344    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-credential-socket" pod "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" (UID: "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:16.019517 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.019482    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-kube-api-access-k5tr7\" (UniqueName: \"kubernetes.io/projected/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-kube-api-access-k5tr7\") pod \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\" (UID: \"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90\") "
Oct 30 22:38:16.019626 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.019547    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-workload-socket" pod "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" (UID: "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:16.019626 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.019580    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istiod-ca-cert" pod "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" (UID: "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:16.020400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.020328    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:16.020400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.020359    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:16.020400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.020374    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:16.020400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.020388    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:16.020400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.020401    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:16.021435 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.021411    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-envoy" pod "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" (UID: "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:16.021588 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.021546    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-podinfo" pod "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" (UID: "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:38:16.021663 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.021588    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-token" pod "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" (UID: "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:16.021923 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.021902    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-kube-api-access-k5tr7" pod "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" (UID: "f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90"). InnerVolumeSpecName "kube-api-access-k5tr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:16.122168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.122118    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:16.122343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.122170    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:16.122343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.122213    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:16.122343 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.122224    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5tr7\" (UniqueName: \"kubernetes.io/projected/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90-kube-api-access-k5tr7\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:16.188743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.188654    4553 generic.go:356] "Generic (PLEG): container finished" podID="f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" containerID="40dc8616ea9fdfb5c82376fe70b5f386111fd57317e5a35fa7875f3b37787045" exitCode=0
Oct 30 22:38:16.188743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.188702    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"
Oct 30 22:38:16.188998 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.188740    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x" event={"ID":"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90","Type":"ContainerDied","Data":"40dc8616ea9fdfb5c82376fe70b5f386111fd57317e5a35fa7875f3b37787045"}
Oct 30 22:38:16.188998 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.188802    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x" event={"ID":"f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90","Type":"ContainerDied","Data":"72f0b20d702fc6e7a2671bd98fc5d65c339dd66ec5278ea5eb9222e43004b238"}
Oct 30 22:38:16.188998 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.188832    4553 scope.go:122] "RemoveContainer" containerID="40dc8616ea9fdfb5c82376fe70b5f386111fd57317e5a35fa7875f3b37787045"
Oct 30 22:38:16.201004 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.200986    4553 scope.go:122] "RemoveContainer" containerID="40dc8616ea9fdfb5c82376fe70b5f386111fd57317e5a35fa7875f3b37787045"
Oct 30 22:38:16.201349 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:38:16.201323    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40dc8616ea9fdfb5c82376fe70b5f386111fd57317e5a35fa7875f3b37787045\": container with ID starting with 40dc8616ea9fdfb5c82376fe70b5f386111fd57317e5a35fa7875f3b37787045 not found: ID does not exist" containerID="40dc8616ea9fdfb5c82376fe70b5f386111fd57317e5a35fa7875f3b37787045"
Oct 30 22:38:16.201411 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.201362    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40dc8616ea9fdfb5c82376fe70b5f386111fd57317e5a35fa7875f3b37787045"} err="failed to get container status \"40dc8616ea9fdfb5c82376fe70b5f386111fd57317e5a35fa7875f3b37787045\": rpc error: code = NotFound desc = could not find container \"40dc8616ea9fdfb5c82376fe70b5f386111fd57317e5a35fa7875f3b37787045\": container with ID starting with 40dc8616ea9fdfb5c82376fe70b5f386111fd57317e5a35fa7875f3b37787045 not found: ID does not exist"
Oct 30 22:38:16.356680 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.356648    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"]
Oct 30 22:38:16.360592 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:16.360566    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-bxr9x"]
Oct 30 22:38:17.394565 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:17.394530    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" path="/var/lib/kubelet/pods/f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90/volumes"
Oct 30 22:38:21.667640 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:21.667602    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:21.680507 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:21.680478    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:22.240505 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:38:22.240472    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:38:28.202996 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:38:28.202961    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:38:58.204198 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:38:58.204157    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:39:01.039549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:01.039501    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"]
Oct 30 22:39:06.040253 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.040210    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t" podUID="da90d52a-d740-441b-a300-209ba4d63323" containerName="payload-processing" containerID="cri-o://cee9f97bcf1eec8823b7f7b2706abf1d491352beda04ae3534ba20e09d5c693f" gracePeriod=25
Oct 30 22:39:06.321314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.321287    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"
Oct 30 22:39:06.497072 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.497037    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da90d52a-d740-441b-a300-209ba4d63323-kube-api-access-lnlwl\" (UniqueName: \"kubernetes.io/projected/da90d52a-d740-441b-a300-209ba4d63323-kube-api-access-lnlwl\") pod \"da90d52a-d740-441b-a300-209ba4d63323\" (UID: \"da90d52a-d740-441b-a300-209ba4d63323\") "
Oct 30 22:39:06.497306 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.497252    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da90d52a-d740-441b-a300-209ba4d63323-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da90d52a-d740-441b-a300-209ba4d63323-plugins-config-volume\") pod \"da90d52a-d740-441b-a300-209ba4d63323\" (UID: \"da90d52a-d740-441b-a300-209ba4d63323\") "
Oct 30 22:39:06.497672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.497639    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da90d52a-d740-441b-a300-209ba4d63323-plugins-config-volume" pod "da90d52a-d740-441b-a300-209ba4d63323" (UID: "da90d52a-d740-441b-a300-209ba4d63323"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:06.497822 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.497779    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da90d52a-d740-441b-a300-209ba4d63323-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:06.499567 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.499542    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da90d52a-d740-441b-a300-209ba4d63323-kube-api-access-lnlwl" pod "da90d52a-d740-441b-a300-209ba4d63323" (UID: "da90d52a-d740-441b-a300-209ba4d63323"). InnerVolumeSpecName "kube-api-access-lnlwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:06.569764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.569675    4553 generic.go:356] "Generic (PLEG): container finished" podID="da90d52a-d740-441b-a300-209ba4d63323" containerID="cee9f97bcf1eec8823b7f7b2706abf1d491352beda04ae3534ba20e09d5c693f" exitCode=0
Oct 30 22:39:06.569764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.569728    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"
Oct 30 22:39:06.569977 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.569763    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t" event={"ID":"da90d52a-d740-441b-a300-209ba4d63323","Type":"ContainerDied","Data":"cee9f97bcf1eec8823b7f7b2706abf1d491352beda04ae3534ba20e09d5c693f"}
Oct 30 22:39:06.569977 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.569838    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t" event={"ID":"da90d52a-d740-441b-a300-209ba4d63323","Type":"ContainerDied","Data":"f8526491f47daf5a17ec9b41c4150e78f7f131cdac54a687528d933e9a79008d"}
Oct 30 22:39:06.569977 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.569868    4553 scope.go:122] "RemoveContainer" containerID="cee9f97bcf1eec8823b7f7b2706abf1d491352beda04ae3534ba20e09d5c693f"
Oct 30 22:39:06.580924 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.580898    4553 scope.go:122] "RemoveContainer" containerID="cee9f97bcf1eec8823b7f7b2706abf1d491352beda04ae3534ba20e09d5c693f"
Oct 30 22:39:06.581288 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:39:06.581259    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee9f97bcf1eec8823b7f7b2706abf1d491352beda04ae3534ba20e09d5c693f\": container with ID starting with cee9f97bcf1eec8823b7f7b2706abf1d491352beda04ae3534ba20e09d5c693f not found: ID does not exist" containerID="cee9f97bcf1eec8823b7f7b2706abf1d491352beda04ae3534ba20e09d5c693f"
Oct 30 22:39:06.581382 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.581300    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee9f97bcf1eec8823b7f7b2706abf1d491352beda04ae3534ba20e09d5c693f"} err="failed to get container status \"cee9f97bcf1eec8823b7f7b2706abf1d491352beda04ae3534ba20e09d5c693f\": rpc error: code = NotFound desc = could not find container \"cee9f97bcf1eec8823b7f7b2706abf1d491352beda04ae3534ba20e09d5c693f\": container with ID starting with cee9f97bcf1eec8823b7f7b2706abf1d491352beda04ae3534ba20e09d5c693f not found: ID does not exist"
Oct 30 22:39:06.599823 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.599752    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnlwl\" (UniqueName: \"kubernetes.io/projected/da90d52a-d740-441b-a300-209ba4d63323-kube-api-access-lnlwl\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:06.738107 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.738070    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"]
Oct 30 22:39:06.745001 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:06.744971    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9651e9-65db7c4bf5-nk44t"]
Oct 30 22:39:07.394691 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:07.394654    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da90d52a-d740-441b-a300-209ba4d63323" path="/var/lib/kubelet/pods/da90d52a-d740-441b-a300-209ba4d63323/volumes"
Oct 30 22:39:16.286438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:16.286394    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"]
Oct 30 22:39:18.439540 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.439502    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"]
Oct 30 22:39:18.440198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.440182    4553 cpu_manager.go:414] "containerMap: removing container" podUID="da90d52a-d740-441b-a300-209ba4d63323" containerName="payload-processing"
Oct 30 22:39:18.440255 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.440199    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da90d52a-d740-441b-a300-209ba4d63323" containerName="payload-processing"
Oct 30 22:39:18.440255 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.440236    4553 cpu_manager.go:414] "containerMap: removing container" podUID="f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" containerName="istio-proxy"
Oct 30 22:39:18.440255 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.440241    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" containerName="istio-proxy"
Oct 30 22:39:18.440428 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.440349    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="da90d52a-d740-441b-a300-209ba4d63323" containerName="payload-processing"
Oct 30 22:39:18.440428 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.440365    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1dc2e21-d5fb-4dc6-895d-fb651fa2ed90" containerName="istio-proxy"
Oct 30 22:39:18.445445 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.445425    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.502723 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.502683    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"]
Oct 30 22:39:18.548438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.548405    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-89cf28\"/\"kube-root-ca.crt\""
Oct 30 22:39:18.548438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.548405    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-89cf28\"/\"openshift-service-ca.crt\""
Oct 30 22:39:18.548936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.548909    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-89cf28\"/\"default-dockercfg-995mh\""
Oct 30 22:39:18.549065 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.548951    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-89cf28\"/\"auth-test-model-89cf28-kserve-self-signed-certs\""
Oct 30 22:39:18.570829 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.570800    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-home\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.570982 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.570920    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-tmp-dir\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.571024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.570979    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3f457a-83fb-4bcd-a884-d9a988379046-tls-certs\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.571097 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.571085    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-model-cache\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.571185 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.571126    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt67j\" (UniqueName: \"kubernetes.io/projected/0f3f457a-83fb-4bcd-a884-d9a988379046-kube-api-access-xt67j\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.571261 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.571247    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-dshm\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.672942 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.672903    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-model-cache\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.672942 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.672941    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt67j\" (UniqueName: \"kubernetes.io/projected/0f3f457a-83fb-4bcd-a884-d9a988379046-kube-api-access-xt67j\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.673284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.672966    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-dshm\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.673284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.673045    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-home\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.673284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.673104    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-tmp-dir\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.673284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.673197    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3f457a-83fb-4bcd-a884-d9a988379046-tls-certs\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.673509 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.673435    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-model-cache\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.673509 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.673498    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-home\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.673771 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.673751    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-tmp-dir\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.675508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.675481    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-dshm\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.675811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.675788    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3f457a-83fb-4bcd-a884-d9a988379046-tls-certs\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.735624 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.735548    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt67j\" (UniqueName: \"kubernetes.io/projected/0f3f457a-83fb-4bcd-a884-d9a988379046-kube-api-access-xt67j\") pod \"auth-test-model-89cf28-kserve-6dff76578f-6h7r9\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") " pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:18.758856 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:18.758822    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:19.147243 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:19.147215    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"]
Oct 30 22:39:19.148477 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:39:19.148443    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f3f457a_83fb_4bcd_a884_d9a988379046.slice/crio-27289f1632468512804285ce64c10f58421322091e674a3079968a0964c0a3aa WatchSource:0}: Error finding container 27289f1632468512804285ce64c10f58421322091e674a3079968a0964c0a3aa: Status 404 returned error can't find the container with id 27289f1632468512804285ce64c10f58421322091e674a3079968a0964c0a3aa
Oct 30 22:39:19.658499 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:19.658461    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9" event={"ID":"0f3f457a-83fb-4bcd-a884-d9a988379046","Type":"ContainerStarted","Data":"2329eccb1d9a8a74e2078baa52a15fc46a64f3648c5f3847dc6f4cf16d935808"}
Oct 30 22:39:19.658982 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:19.658508    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9" event={"ID":"0f3f457a-83fb-4bcd-a884-d9a988379046","Type":"ContainerStarted","Data":"27289f1632468512804285ce64c10f58421322091e674a3079968a0964c0a3aa"}
Oct 30 22:39:19.759971 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:19.759929    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:19.780636 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:19.780579    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9" podStartSLOduration=1.780562177 podStartE2EDuration="1.780562177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:19.779912434 +0000 UTC m=+3410.514275478" watchObservedRunningTime="2026-10-30 22:39:19.780562177 +0000 UTC m=+3410.514925211"
Oct 30 22:39:28.203217 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:39:28.203175    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:39:28.759798 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:28.759766    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:28.773816 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:28.773784    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:29.738056 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:29.738024    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:39:31.308467 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.308421    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d" podUID="e12162f0-3b43-4bcf-bd0a-c4298b61dbd9" containerName="main" containerID="cri-o://be9bfbc3f1cd3b5161db603626d852956c5ffb904926afccecf6222d81ac4368" gracePeriod=45
Oct 30 22:39:31.581688 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.581655    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:39:31.655520 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.655484    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-home\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-home\") pod \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") "
Oct 30 22:39:31.655756 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.655700    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-model-cache\") pod \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") "
Oct 30 22:39:31.655836 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.655763    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-tls-certs\" (UniqueName: \"kubernetes.io/secret/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-tls-certs\") pod \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") "
Oct 30 22:39:31.655836 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.655785    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-home" pod "e12162f0-3b43-4bcf-bd0a-c4298b61dbd9" (UID: "e12162f0-3b43-4bcf-bd0a-c4298b61dbd9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:31.655836 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.655811    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-kube-api-access-9t6w4\" (UniqueName: \"kubernetes.io/projected/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-kube-api-access-9t6w4\") pod \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") "
Oct 30 22:39:31.656016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.655947    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-dshm\") pod \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") "
Oct 30 22:39:31.656016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.655979    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-tmp-dir\") pod \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\" (UID: \"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9\") "
Oct 30 22:39:31.656016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.655998    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-model-cache" pod "e12162f0-3b43-4bcf-bd0a-c4298b61dbd9" (UID: "e12162f0-3b43-4bcf-bd0a-c4298b61dbd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:31.656794 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.656579    4553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-home\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:31.656794 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.656671    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-tmp-dir" pod "e12162f0-3b43-4bcf-bd0a-c4298b61dbd9" (UID: "e12162f0-3b43-4bcf-bd0a-c4298b61dbd9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:31.656794 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.656767    4553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-model-cache\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:31.658837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.658813    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-kube-api-access-9t6w4" pod "e12162f0-3b43-4bcf-bd0a-c4298b61dbd9" (UID: "e12162f0-3b43-4bcf-bd0a-c4298b61dbd9"). InnerVolumeSpecName "kube-api-access-9t6w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:31.658837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.658822    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-dshm" pod "e12162f0-3b43-4bcf-bd0a-c4298b61dbd9" (UID: "e12162f0-3b43-4bcf-bd0a-c4298b61dbd9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:31.658991 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.658813    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-tls-certs" pod "e12162f0-3b43-4bcf-bd0a-c4298b61dbd9" (UID: "e12162f0-3b43-4bcf-bd0a-c4298b61dbd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:39:31.745244 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.745207    4553 generic.go:356] "Generic (PLEG): container finished" podID="e12162f0-3b43-4bcf-bd0a-c4298b61dbd9" containerID="be9bfbc3f1cd3b5161db603626d852956c5ffb904926afccecf6222d81ac4368" exitCode=0
Oct 30 22:39:31.745423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.745256    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"
Oct 30 22:39:31.745423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.745288    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d" event={"ID":"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9","Type":"ContainerDied","Data":"be9bfbc3f1cd3b5161db603626d852956c5ffb904926afccecf6222d81ac4368"}
Oct 30 22:39:31.745423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.745330    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d" event={"ID":"e12162f0-3b43-4bcf-bd0a-c4298b61dbd9","Type":"ContainerDied","Data":"712b97ea66a133a306d851dc60753f3ed49cd187e5501f243635d5f4b6a9f34c"}
Oct 30 22:39:31.745423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.745347    4553 scope.go:122] "RemoveContainer" containerID="be9bfbc3f1cd3b5161db603626d852956c5ffb904926afccecf6222d81ac4368"
Oct 30 22:39:31.756342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.756322    4553 scope.go:122] "RemoveContainer" containerID="be9bfbc3f1cd3b5161db603626d852956c5ffb904926afccecf6222d81ac4368"
Oct 30 22:39:31.756635 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:39:31.756613    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9bfbc3f1cd3b5161db603626d852956c5ffb904926afccecf6222d81ac4368\": container with ID starting with be9bfbc3f1cd3b5161db603626d852956c5ffb904926afccecf6222d81ac4368 not found: ID does not exist" containerID="be9bfbc3f1cd3b5161db603626d852956c5ffb904926afccecf6222d81ac4368"
Oct 30 22:39:31.756690 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.756645    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9bfbc3f1cd3b5161db603626d852956c5ffb904926afccecf6222d81ac4368"} err="failed to get container status \"be9bfbc3f1cd3b5161db603626d852956c5ffb904926afccecf6222d81ac4368\": rpc error: code = NotFound desc = could not find container \"be9bfbc3f1cd3b5161db603626d852956c5ffb904926afccecf6222d81ac4368\": container with ID starting with be9bfbc3f1cd3b5161db603626d852956c5ffb904926afccecf6222d81ac4368 not found: ID does not exist"
Oct 30 22:39:31.758517 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.758491    4553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-tls-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:31.758617 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.758518    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t6w4\" (UniqueName: \"kubernetes.io/projected/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-kube-api-access-9t6w4\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:31.758617 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.758532    4553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-dshm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:31.758617 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.758545    4553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9-tmp-dir\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:31.913222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.913191    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"]
Oct 30 22:39:31.918797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:31.918772    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-9651e9/explicit-model-9651e9-kserve-7dcc79f48f-qs98d"]
Oct 30 22:39:33.394090 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:33.394058    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12162f0-3b43-4bcf-bd0a-c4298b61dbd9" path="/var/lib/kubelet/pods/e12162f0-3b43-4bcf-bd0a-c4298b61dbd9/volumes"
Oct 30 22:39:35.642010 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:35.641973    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"]
Oct 30 22:39:35.642600 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:35.642212    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv" podUID="aff22a9f-32c6-4373-8d9d-62f6ce177851" containerName="istio-proxy" containerID="cri-o://ec80e0f5661e87c1cc769fc7604e6904ecbe360165c2579a26660ca992ab223b" gracePeriod=30
Oct 30 22:39:40.814223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:40.814189    4553 generic.go:356] "Generic (PLEG): container finished" podID="aff22a9f-32c6-4373-8d9d-62f6ce177851" containerID="ec80e0f5661e87c1cc769fc7604e6904ecbe360165c2579a26660ca992ab223b" exitCode=0
Oct 30 22:39:40.814697 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:40.814263    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv" event={"ID":"aff22a9f-32c6-4373-8d9d-62f6ce177851","Type":"ContainerDied","Data":"ec80e0f5661e87c1cc769fc7604e6904ecbe360165c2579a26660ca992ab223b"}
Oct 30 22:39:40.814697 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:40.814348    4553 scope.go:122] "RemoveContainer" containerID="02144e37d2ad8dce8951e78a398c810f112810d14266d2e9dbc15cb2bccfad8f"
Oct 30 22:39:40.925624 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:40.925551    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:39:41.087646 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.087609    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-podinfo\") pod \"aff22a9f-32c6-4373-8d9d-62f6ce177851\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") "
Oct 30 22:39:41.087879 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.087717    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-workload-certs\") pod \"aff22a9f-32c6-4373-8d9d-62f6ce177851\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") "
Oct 30 22:39:41.087879 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.087757    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff22a9f-32c6-4373-8d9d-62f6ce177851-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff22a9f-32c6-4373-8d9d-62f6ce177851-istiod-ca-cert\") pod \"aff22a9f-32c6-4373-8d9d-62f6ce177851\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") "
Oct 30 22:39:41.087879 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.087851    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-envoy\") pod \"aff22a9f-32c6-4373-8d9d-62f6ce177851\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") "
Oct 30 22:39:41.088059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.087888    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-data\") pod \"aff22a9f-32c6-4373-8d9d-62f6ce177851\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") "
Oct 30 22:39:41.088059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.087961    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-token\" (UniqueName: \"kubernetes.io/projected/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-token\") pod \"aff22a9f-32c6-4373-8d9d-62f6ce177851\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") "
Oct 30 22:39:41.088059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.088023    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-workload-certs" pod "aff22a9f-32c6-4373-8d9d-62f6ce177851" (UID: "aff22a9f-32c6-4373-8d9d-62f6ce177851"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:41.088059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.088048    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-workload-socket\") pod \"aff22a9f-32c6-4373-8d9d-62f6ce177851\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") "
Oct 30 22:39:41.088311 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.088114    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff22a9f-32c6-4373-8d9d-62f6ce177851-istiod-ca-cert" pod "aff22a9f-32c6-4373-8d9d-62f6ce177851" (UID: "aff22a9f-32c6-4373-8d9d-62f6ce177851"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:41.088311 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.088233    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-workload-socket" pod "aff22a9f-32c6-4373-8d9d-62f6ce177851" (UID: "aff22a9f-32c6-4373-8d9d-62f6ce177851"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:41.088311 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.088238    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff22a9f-32c6-4373-8d9d-62f6ce177851-kube-api-access-mmzdk\" (UniqueName: \"kubernetes.io/projected/aff22a9f-32c6-4373-8d9d-62f6ce177851-kube-api-access-mmzdk\") pod \"aff22a9f-32c6-4373-8d9d-62f6ce177851\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") "
Oct 30 22:39:41.088311 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.088252    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-data" pod "aff22a9f-32c6-4373-8d9d-62f6ce177851" (UID: "aff22a9f-32c6-4373-8d9d-62f6ce177851"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:41.088311 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.088280    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-credential-socket\") pod \"aff22a9f-32c6-4373-8d9d-62f6ce177851\" (UID: \"aff22a9f-32c6-4373-8d9d-62f6ce177851\") "
Oct 30 22:39:41.088817 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.088545    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-credential-socket" pod "aff22a9f-32c6-4373-8d9d-62f6ce177851" (UID: "aff22a9f-32c6-4373-8d9d-62f6ce177851"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:41.088955 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.088934    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:41.089033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.088958    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:41.089033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.088967    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff22a9f-32c6-4373-8d9d-62f6ce177851-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:41.089033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.088977    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:41.089033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.088985    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:41.090284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.090262    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-podinfo" pod "aff22a9f-32c6-4373-8d9d-62f6ce177851" (UID: "aff22a9f-32c6-4373-8d9d-62f6ce177851"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:39:41.090284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.090269    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-envoy" pod "aff22a9f-32c6-4373-8d9d-62f6ce177851" (UID: "aff22a9f-32c6-4373-8d9d-62f6ce177851"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:41.090926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.090908    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff22a9f-32c6-4373-8d9d-62f6ce177851-kube-api-access-mmzdk" pod "aff22a9f-32c6-4373-8d9d-62f6ce177851" (UID: "aff22a9f-32c6-4373-8d9d-62f6ce177851"). InnerVolumeSpecName "kube-api-access-mmzdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:41.090995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.090932    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-token" pod "aff22a9f-32c6-4373-8d9d-62f6ce177851" (UID: "aff22a9f-32c6-4373-8d9d-62f6ce177851"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:41.190278 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.190193    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:41.190278 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.190220    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:41.190278 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.190231    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff22a9f-32c6-4373-8d9d-62f6ce177851-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:41.190278 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.190240    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmzdk\" (UniqueName: \"kubernetes.io/projected/aff22a9f-32c6-4373-8d9d-62f6ce177851-kube-api-access-mmzdk\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:39:41.822348 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.822321    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"
Oct 30 22:39:41.822867 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.822345    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv" event={"ID":"aff22a9f-32c6-4373-8d9d-62f6ce177851","Type":"ContainerDied","Data":"45b0f7f59716806a78f3ce3ebfaf4a5912ab39b9be67e843313707451a3d52dc"}
Oct 30 22:39:41.822867 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:41.822407    4553 scope.go:122] "RemoveContainer" containerID="ec80e0f5661e87c1cc769fc7604e6904ecbe360165c2579a26660ca992ab223b"
Oct 30 22:39:42.040839 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:42.040801    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"]
Oct 30 22:39:42.091269 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:42.091241    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-9651e9-openshift-default-569445c5c-2gjbv"]
Oct 30 22:39:43.394401 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:39:43.394369    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff22a9f-32c6-4373-8d9d-62f6ce177851" path="/var/lib/kubelet/pods/aff22a9f-32c6-4373-8d9d-62f6ce177851/volumes"
Oct 30 22:39:58.201973 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:39:58.201935    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:40:28.205955 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:40:28.205891    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:40:34.729935 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.729897    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"]
Oct 30 22:40:34.730695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.730674    4553 cpu_manager.go:414] "containerMap: removing container" podUID="e12162f0-3b43-4bcf-bd0a-c4298b61dbd9" containerName="main"
Oct 30 22:40:34.730695 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.730696    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12162f0-3b43-4bcf-bd0a-c4298b61dbd9" containerName="main"
Oct 30 22:40:34.730877 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.730730    4553 cpu_manager.go:414] "containerMap: removing container" podUID="aff22a9f-32c6-4373-8d9d-62f6ce177851" containerName="istio-proxy"
Oct 30 22:40:34.730877 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.730738    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff22a9f-32c6-4373-8d9d-62f6ce177851" containerName="istio-proxy"
Oct 30 22:40:34.730877 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.730763    4553 cpu_manager.go:414] "containerMap: removing container" podUID="aff22a9f-32c6-4373-8d9d-62f6ce177851" containerName="istio-proxy"
Oct 30 22:40:34.730877 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.730769    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff22a9f-32c6-4373-8d9d-62f6ce177851" containerName="istio-proxy"
Oct 30 22:40:34.731072 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.730898    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12162f0-3b43-4bcf-bd0a-c4298b61dbd9" containerName="main"
Oct 30 22:40:34.731072 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.730918    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff22a9f-32c6-4373-8d9d-62f6ce177851" containerName="istio-proxy"
Oct 30 22:40:34.731072 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.730934    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff22a9f-32c6-4373-8d9d-62f6ce177851" containerName="istio-proxy"
Oct 30 22:40:34.735241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.735220    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.793361 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.793310    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-dshm\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.793537 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.793379    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-home\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.793537 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.793457    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd7c427b-c8e5-43e1-ace7-125988403351-tls-certs\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.793537 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.793494    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flcfn\" (UniqueName: \"kubernetes.io/projected/fd7c427b-c8e5-43e1-ace7-125988403351-kube-api-access-flcfn\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.793672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.793574    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-model-cache\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.793672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.793651    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-tmp-dir\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.838808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.838745    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6eae99\"/\"openshift-service-ca.crt\""
Oct 30 22:40:34.839017 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.838863    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6eae99\"/\"kube-root-ca.crt\""
Oct 30 22:40:34.839017 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.838868    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6eae99\"/\"noten-model-6eae99-kserve-self-signed-certs\""
Oct 30 22:40:34.839017 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.838867    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6eae99\"/\"default-dockercfg-gddfl\""
Oct 30 22:40:34.840297 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.840271    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"]
Oct 30 22:40:34.895501 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.895468    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-model-cache\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.895730 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.895527    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-tmp-dir\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.895730 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.895592    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-dshm\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.895730 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.895638    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-home\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.895730 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.895701    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd7c427b-c8e5-43e1-ace7-125988403351-tls-certs\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.895994 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.895738    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flcfn\" (UniqueName: \"kubernetes.io/projected/fd7c427b-c8e5-43e1-ace7-125988403351-kube-api-access-flcfn\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.895994 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.895888    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-model-cache\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.895994 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.895965    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-tmp-dir\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.896191 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.896023    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-home\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.898825 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.898796    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-dshm\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.899015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.898998    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd7c427b-c8e5-43e1-ace7-125988403351-tls-certs\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:34.953739 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:34.953698    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flcfn\" (UniqueName: \"kubernetes.io/projected/fd7c427b-c8e5-43e1-ace7-125988403351-kube-api-access-flcfn\") pod \"noten-model-6eae99-kserve-75456b68b5-x6l5w\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") " pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:35.049671 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:35.049581    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:35.431622 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:35.431596    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"]
Oct 30 22:40:35.432250 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:40:35.432226    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd7c427b_c8e5_43e1_ace7_125988403351.slice/crio-e9ad87704f65cf2f2d36537bd547d3c6bac1d70895508dc16716a25f3eb76218 WatchSource:0}: Error finding container e9ad87704f65cf2f2d36537bd547d3c6bac1d70895508dc16716a25f3eb76218: Status 404 returned error can't find the container with id e9ad87704f65cf2f2d36537bd547d3c6bac1d70895508dc16716a25f3eb76218
Oct 30 22:40:36.177473 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:36.177438    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w" event={"ID":"fd7c427b-c8e5-43e1-ace7-125988403351","Type":"ContainerStarted","Data":"ab5bf84e361421f9256dc24edbbe1e65a9213ba2dbf3a752e63453d6604d546f"}
Oct 30 22:40:36.177895 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:36.177487    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w" event={"ID":"fd7c427b-c8e5-43e1-ace7-125988403351","Type":"ContainerStarted","Data":"e9ad87704f65cf2f2d36537bd547d3c6bac1d70895508dc16716a25f3eb76218"}
Oct 30 22:40:36.293861 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:36.293806    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w" podStartSLOduration=2.293793395 podStartE2EDuration="2.293793395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:36.293038771 +0000 UTC m=+3487.027401803" watchObservedRunningTime="2026-10-30 22:40:36.293793395 +0000 UTC m=+3487.028156425"
Oct 30 22:40:37.050596 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:37.050557    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:45.050717 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:45.050681    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:45.063426 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:45.063399    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:45.255612 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:40:45.255584    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:40:58.203771 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:40:58.203724    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:41:05.503941 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.503909    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"]
Oct 30 22:41:05.508243 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.508212    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.565579 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.565539    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e450a\"/\"openshift-service-ca.crt\""
Oct 30 22:41:05.569889 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.569853    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"]
Oct 30 22:41:05.592679 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.592645    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-home\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.592679 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.592682    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-dshm\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.592973 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.592726    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-model-cache\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.592973 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.592837    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pzv8\" (UniqueName: \"kubernetes.io/projected/d328e0bf-ad1f-4d08-a16d-f3e012187a85-kube-api-access-2pzv8\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.592973 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.592912    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-tmp-dir\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.593184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.593070    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d328e0bf-ad1f-4d08-a16d-f3e012187a85-tls-certs\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.616000 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.615968    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e450a\"/\"kube-root-ca.crt\""
Oct 30 22:41:05.616189 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.616037    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e450a\"/\"default-dockercfg-kjz9j\""
Oct 30 22:41:05.616189 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.616078    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0e450a\"/\"auth-test-model-0e450a-kserve-self-signed-certs\""
Oct 30 22:41:05.693953 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.693921    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d328e0bf-ad1f-4d08-a16d-f3e012187a85-tls-certs\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.694165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.694036    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-home\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.694165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.694056    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-dshm\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.694165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.694087    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-model-cache\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.694165 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.694113    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pzv8\" (UniqueName: \"kubernetes.io/projected/d328e0bf-ad1f-4d08-a16d-f3e012187a85-kube-api-access-2pzv8\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.694379 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.694227    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-tmp-dir\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.694524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.694503    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-model-cache\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.694605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.694586    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-home\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.694664 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.694636    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-tmp-dir\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.696582 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.696558    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-dshm\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.696705 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.696687    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d328e0bf-ad1f-4d08-a16d-f3e012187a85-tls-certs\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.753585 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.753556    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pzv8\" (UniqueName: \"kubernetes.io/projected/d328e0bf-ad1f-4d08-a16d-f3e012187a85-kube-api-access-2pzv8\") pod \"auth-test-model-0e450a-kserve-6f5bf5d565-lftc9\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") " pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:05.824218 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:05.824112    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:06.209061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:06.209028    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"]
Oct 30 22:41:06.210125 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:41:06.210097    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd328e0bf_ad1f_4d08_a16d_f3e012187a85.slice/crio-9f8f5b61681f648f2246d62105a3b279c270b1b4d278729d8461a9195dbc4307 WatchSource:0}: Error finding container 9f8f5b61681f648f2246d62105a3b279c270b1b4d278729d8461a9195dbc4307: Status 404 returned error can't find the container with id 9f8f5b61681f648f2246d62105a3b279c270b1b4d278729d8461a9195dbc4307
Oct 30 22:41:06.394307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:06.394270    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9" event={"ID":"d328e0bf-ad1f-4d08-a16d-f3e012187a85","Type":"ContainerStarted","Data":"975fe2f536fed5569eb79a621f462b32c0ab52728864a957c1fe8a16b602d46d"}
Oct 30 22:41:06.394509 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:06.394325    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9" event={"ID":"d328e0bf-ad1f-4d08-a16d-f3e012187a85","Type":"ContainerStarted","Data":"9f8f5b61681f648f2246d62105a3b279c270b1b4d278729d8461a9195dbc4307"}
Oct 30 22:41:06.514291 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:06.514185    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9" podStartSLOduration=1.514169229 podStartE2EDuration="1.514169229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:06.510822491 +0000 UTC m=+3517.245185524" watchObservedRunningTime="2026-10-30 22:41:06.514169229 +0000 UTC m=+3517.248532259"
Oct 30 22:41:06.824318 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:06.824237    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:15.825396 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:15.825366    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:15.843244 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:15.843211    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:16.473789 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:16.473757    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:41:21.334680 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:21.334637    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"]
Oct 30 22:41:25.014495 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:25.014457    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"]
Oct 30 22:41:25.014924 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:25.014673    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68" podUID="769f2ed9-d041-4d27-ba11-3a6ad3ebf455" containerName="istio-proxy" containerID="cri-o://a499ae04bc87bd9a0c75b20a624355f584168304cdeba0054274b897c38e8f34" gracePeriod=30
Oct 30 22:41:28.202706 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:41:28.202666    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:41:29.545711 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:29.545670    4553 patch_prober.go:29] interesting pod/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:41:29.546230 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:29.545745    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68" podUID="769f2ed9-d041-4d27-ba11-3a6ad3ebf455" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:41:30.316183 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.316155    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:41:30.437632 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.437597    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-workload-socket\") pod \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") "
Oct 30 22:41:30.437849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.437642    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istiod-ca-cert\") pod \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") "
Oct 30 22:41:30.437849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.437724    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-data\") pod \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") "
Oct 30 22:41:30.437976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.437875    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-envoy\") pod \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") "
Oct 30 22:41:30.437976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.437900    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-workload-socket" pod "769f2ed9-d041-4d27-ba11-3a6ad3ebf455" (UID: "769f2ed9-d041-4d27-ba11-3a6ad3ebf455"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:30.437976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.437953    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-token\" (UniqueName: \"kubernetes.io/projected/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-token\") pod \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") "
Oct 30 22:41:30.438184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.437989    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-podinfo\") pod \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") "
Oct 30 22:41:30.438184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.438007    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-data" pod "769f2ed9-d041-4d27-ba11-3a6ad3ebf455" (UID: "769f2ed9-d041-4d27-ba11-3a6ad3ebf455"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:30.438184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.438040    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-workload-certs\") pod \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") "
Oct 30 22:41:30.438184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.438078    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istiod-ca-cert" pod "769f2ed9-d041-4d27-ba11-3a6ad3ebf455" (UID: "769f2ed9-d041-4d27-ba11-3a6ad3ebf455"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:30.438184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.438112    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-credential-socket\") pod \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") "
Oct 30 22:41:30.438451 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.438209    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-kube-api-access-htsr4\" (UniqueName: \"kubernetes.io/projected/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-kube-api-access-htsr4\") pod \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\" (UID: \"769f2ed9-d041-4d27-ba11-3a6ad3ebf455\") "
Oct 30 22:41:30.438451 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.438364    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-workload-certs" pod "769f2ed9-d041-4d27-ba11-3a6ad3ebf455" (UID: "769f2ed9-d041-4d27-ba11-3a6ad3ebf455"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:30.438545 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.438484    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-credential-socket" pod "769f2ed9-d041-4d27-ba11-3a6ad3ebf455" (UID: "769f2ed9-d041-4d27-ba11-3a6ad3ebf455"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:30.438890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.438800    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.438890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.438829    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.438890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.438840    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.438890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.438849    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.438890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.438858    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.440416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.440389    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-token" pod "769f2ed9-d041-4d27-ba11-3a6ad3ebf455" (UID: "769f2ed9-d041-4d27-ba11-3a6ad3ebf455"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:30.440549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.440530    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-envoy" pod "769f2ed9-d041-4d27-ba11-3a6ad3ebf455" (UID: "769f2ed9-d041-4d27-ba11-3a6ad3ebf455"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:30.440862 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.440835    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-podinfo" pod "769f2ed9-d041-4d27-ba11-3a6ad3ebf455" (UID: "769f2ed9-d041-4d27-ba11-3a6ad3ebf455"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:41:30.440952 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.440864    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-kube-api-access-htsr4" pod "769f2ed9-d041-4d27-ba11-3a6ad3ebf455" (UID: "769f2ed9-d041-4d27-ba11-3a6ad3ebf455"). InnerVolumeSpecName "kube-api-access-htsr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:30.540643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.540560    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htsr4\" (UniqueName: \"kubernetes.io/projected/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-kube-api-access-htsr4\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.540643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.540589    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.540643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.540598    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.540643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.540608    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769f2ed9-d041-4d27-ba11-3a6ad3ebf455-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:30.557558 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.557526    4553 generic.go:356] "Generic (PLEG): container finished" podID="769f2ed9-d041-4d27-ba11-3a6ad3ebf455" containerID="a499ae04bc87bd9a0c75b20a624355f584168304cdeba0054274b897c38e8f34" exitCode=0
Oct 30 22:41:30.558037 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.557583    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"
Oct 30 22:41:30.558037 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.557610    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68" event={"ID":"769f2ed9-d041-4d27-ba11-3a6ad3ebf455","Type":"ContainerDied","Data":"a499ae04bc87bd9a0c75b20a624355f584168304cdeba0054274b897c38e8f34"}
Oct 30 22:41:30.558037 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.557680    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68" event={"ID":"769f2ed9-d041-4d27-ba11-3a6ad3ebf455","Type":"ContainerDied","Data":"507543d6295075b644531f3abcfea7c57e8e7f084b40fd8a820fea7bc8c8ea4b"}
Oct 30 22:41:30.558037 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.557714    4553 scope.go:122] "RemoveContainer" containerID="a499ae04bc87bd9a0c75b20a624355f584168304cdeba0054274b897c38e8f34"
Oct 30 22:41:30.569395 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.569376    4553 scope.go:122] "RemoveContainer" containerID="a499ae04bc87bd9a0c75b20a624355f584168304cdeba0054274b897c38e8f34"
Oct 30 22:41:30.569711 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:41:30.569690    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a499ae04bc87bd9a0c75b20a624355f584168304cdeba0054274b897c38e8f34\": container with ID starting with a499ae04bc87bd9a0c75b20a624355f584168304cdeba0054274b897c38e8f34 not found: ID does not exist" containerID="a499ae04bc87bd9a0c75b20a624355f584168304cdeba0054274b897c38e8f34"
Oct 30 22:41:30.569790 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.569724    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a499ae04bc87bd9a0c75b20a624355f584168304cdeba0054274b897c38e8f34"} err="failed to get container status \"a499ae04bc87bd9a0c75b20a624355f584168304cdeba0054274b897c38e8f34\": rpc error: code = NotFound desc = could not find container \"a499ae04bc87bd9a0c75b20a624355f584168304cdeba0054274b897c38e8f34\": container with ID starting with a499ae04bc87bd9a0c75b20a624355f584168304cdeba0054274b897c38e8f34 not found: ID does not exist"
Oct 30 22:41:30.732654 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.730368    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"]
Oct 30 22:41:30.735954 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:30.735931    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b8717e-openshift-default-5b5cc794f-d8s68"]
Oct 30 22:41:31.394714 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:31.394671    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769f2ed9-d041-4d27-ba11-3a6ad3ebf455" path="/var/lib/kubelet/pods/769f2ed9-d041-4d27-ba11-3a6ad3ebf455/volumes"
Oct 30 22:41:36.357929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.357885    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w" podUID="fd7c427b-c8e5-43e1-ace7-125988403351" containerName="main" containerID="cri-o://ab5bf84e361421f9256dc24edbbe1e65a9213ba2dbf3a752e63453d6604d546f" gracePeriod=45
Oct 30 22:41:36.601112 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.601081    4553 generic.go:356] "Generic (PLEG): container finished" podID="fd7c427b-c8e5-43e1-ace7-125988403351" containerID="ab5bf84e361421f9256dc24edbbe1e65a9213ba2dbf3a752e63453d6604d546f" exitCode=0
Oct 30 22:41:36.601407 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.601157    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w" event={"ID":"fd7c427b-c8e5-43e1-ace7-125988403351","Type":"ContainerDied","Data":"ab5bf84e361421f9256dc24edbbe1e65a9213ba2dbf3a752e63453d6604d546f"}
Oct 30 22:41:36.641028 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.641003    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:41:36.719815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.719781    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-home\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-home\") pod \"fd7c427b-c8e5-43e1-ace7-125988403351\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") "
Oct 30 22:41:36.720089 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.719941    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-dshm\") pod \"fd7c427b-c8e5-43e1-ace7-125988403351\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") "
Oct 30 22:41:36.720089 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.720013    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7c427b-c8e5-43e1-ace7-125988403351-kube-api-access-flcfn\" (UniqueName: \"kubernetes.io/projected/fd7c427b-c8e5-43e1-ace7-125988403351-kube-api-access-flcfn\") pod \"fd7c427b-c8e5-43e1-ace7-125988403351\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") "
Oct 30 22:41:36.720275 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.720108    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-home" pod "fd7c427b-c8e5-43e1-ace7-125988403351" (UID: "fd7c427b-c8e5-43e1-ace7-125988403351"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:36.720275 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.720248    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-tmp-dir\") pod \"fd7c427b-c8e5-43e1-ace7-125988403351\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") "
Oct 30 22:41:36.720388 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.720295    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd7c427b-c8e5-43e1-ace7-125988403351-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd7c427b-c8e5-43e1-ace7-125988403351-tls-certs\") pod \"fd7c427b-c8e5-43e1-ace7-125988403351\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") "
Oct 30 22:41:36.720388 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.720346    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-model-cache\") pod \"fd7c427b-c8e5-43e1-ace7-125988403351\" (UID: \"fd7c427b-c8e5-43e1-ace7-125988403351\") "
Oct 30 22:41:36.720834 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.720815    4553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-home\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:36.720962 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.720878    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-model-cache" pod "fd7c427b-c8e5-43e1-ace7-125988403351" (UID: "fd7c427b-c8e5-43e1-ace7-125988403351"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:36.721028 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.720976    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-tmp-dir" pod "fd7c427b-c8e5-43e1-ace7-125988403351" (UID: "fd7c427b-c8e5-43e1-ace7-125988403351"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:36.722610 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.722586    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7c427b-c8e5-43e1-ace7-125988403351-kube-api-access-flcfn" pod "fd7c427b-c8e5-43e1-ace7-125988403351" (UID: "fd7c427b-c8e5-43e1-ace7-125988403351"). InnerVolumeSpecName "kube-api-access-flcfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:36.722715 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.722631    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-dshm" pod "fd7c427b-c8e5-43e1-ace7-125988403351" (UID: "fd7c427b-c8e5-43e1-ace7-125988403351"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:36.722715 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.722666    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd7c427b-c8e5-43e1-ace7-125988403351-tls-certs" pod "fd7c427b-c8e5-43e1-ace7-125988403351" (UID: "fd7c427b-c8e5-43e1-ace7-125988403351"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:36.822010 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.821981    4553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-dshm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:36.822010 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.822009    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flcfn\" (UniqueName: \"kubernetes.io/projected/fd7c427b-c8e5-43e1-ace7-125988403351-kube-api-access-flcfn\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:36.822010 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.822018    4553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-tmp-dir\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:36.822314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.822027    4553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd7c427b-c8e5-43e1-ace7-125988403351-tls-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:36.822314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:36.822036    4553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd7c427b-c8e5-43e1-ace7-125988403351-model-cache\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:37.609114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:37.609071    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w" event={"ID":"fd7c427b-c8e5-43e1-ace7-125988403351","Type":"ContainerDied","Data":"e9ad87704f65cf2f2d36537bd547d3c6bac1d70895508dc16716a25f3eb76218"}
Oct 30 22:41:37.609667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:37.609152    4553 scope.go:122] "RemoveContainer" containerID="ab5bf84e361421f9256dc24edbbe1e65a9213ba2dbf3a752e63453d6604d546f"
Oct 30 22:41:37.609667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:37.609079    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"
Oct 30 22:41:37.778219 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:37.778106    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"]
Oct 30 22:41:37.828526 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:37.828489    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-6eae99/noten-model-6eae99-kserve-75456b68b5-x6l5w"]
Oct 30 22:41:39.394432 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:41:39.394398    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7c427b-c8e5-43e1-ace7-125988403351" path="/var/lib/kubelet/pods/fd7c427b-c8e5-43e1-ace7-125988403351/volumes"
Oct 30 22:41:58.203608 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:41:58.203563    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:42:22.365304 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.365257    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"]
Oct 30 22:42:22.367987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.365933    4553 cpu_manager.go:414] "containerMap: removing container" podUID="fd7c427b-c8e5-43e1-ace7-125988403351" containerName="main"
Oct 30 22:42:22.367987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.365946    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7c427b-c8e5-43e1-ace7-125988403351" containerName="main"
Oct 30 22:42:22.367987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.365965    4553 cpu_manager.go:414] "containerMap: removing container" podUID="769f2ed9-d041-4d27-ba11-3a6ad3ebf455" containerName="istio-proxy"
Oct 30 22:42:22.367987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.365971    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769f2ed9-d041-4d27-ba11-3a6ad3ebf455" containerName="istio-proxy"
Oct 30 22:42:22.367987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.366089    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="769f2ed9-d041-4d27-ba11-3a6ad3ebf455" containerName="istio-proxy"
Oct 30 22:42:22.367987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.366107    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7c427b-c8e5-43e1-ace7-125988403351" containerName="main"
Oct 30 22:42:22.369650 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.369629    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"
Oct 30 22:42:22.398227 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.398186    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j6f8\" (UniqueName: \"kubernetes.io/projected/843153d0-0973-45a5-b4bc-063f6ea52968-kube-api-access-7j6f8\") pod \"payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br\" (UID: \"843153d0-0973-45a5-b4bc-063f6ea52968\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"
Oct 30 22:42:22.398410 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.398326    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/843153d0-0973-45a5-b4bc-063f6ea52968-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br\" (UID: \"843153d0-0973-45a5-b4bc-063f6ea52968\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"
Oct 30 22:42:22.426511 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.426477    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"]
Oct 30 22:42:22.473929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.473893    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a6900c\""
Oct 30 22:42:22.499573 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.499512    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/843153d0-0973-45a5-b4bc-063f6ea52968-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br\" (UID: \"843153d0-0973-45a5-b4bc-063f6ea52968\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"
Oct 30 22:42:22.499790 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.499725    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j6f8\" (UniqueName: \"kubernetes.io/projected/843153d0-0973-45a5-b4bc-063f6ea52968-kube-api-access-7j6f8\") pod \"payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br\" (UID: \"843153d0-0973-45a5-b4bc-063f6ea52968\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"
Oct 30 22:42:22.500591 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.500565    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/843153d0-0973-45a5-b4bc-063f6ea52968-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br\" (UID: \"843153d0-0973-45a5-b4bc-063f6ea52968\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"
Oct 30 22:42:22.559643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.559615    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j6f8\" (UniqueName: \"kubernetes.io/projected/843153d0-0973-45a5-b4bc-063f6ea52968-kube-api-access-7j6f8\") pod \"payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br\" (UID: \"843153d0-0973-45a5-b4bc-063f6ea52968\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"
Oct 30 22:42:22.683974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:22.683887    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"
Oct 30 22:42:23.064663 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:23.064627    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"]
Oct 30 22:42:23.065726 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:42:23.065686    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod843153d0_0973_45a5_b4bc_063f6ea52968.slice/crio-207a6cd5e9ae096235862073e6a77d2f255d479615a49455bf207a009498ea55 WatchSource:0}: Error finding container 207a6cd5e9ae096235862073e6a77d2f255d479615a49455bf207a009498ea55: Status 404 returned error can't find the container with id 207a6cd5e9ae096235862073e6a77d2f255d479615a49455bf207a009498ea55
Oct 30 22:42:23.937832 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:23.937290    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br" event={"ID":"843153d0-0973-45a5-b4bc-063f6ea52968","Type":"ContainerStarted","Data":"7a6d92250237c240aaa27d1c9b9cd6556f83e024bb3189f5d2c1e1d6112bb5cf"}
Oct 30 22:42:23.937832 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:23.937375    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br" event={"ID":"843153d0-0973-45a5-b4bc-063f6ea52968","Type":"ContainerStarted","Data":"207a6cd5e9ae096235862073e6a77d2f255d479615a49455bf207a009498ea55"}
Oct 30 22:42:23.939118 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:23.938623    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"
Oct 30 22:42:24.057243 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:24.057179    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br" podStartSLOduration=2.057122025 podStartE2EDuration="2.057122025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:24.055247797 +0000 UTC m=+3594.789610827" watchObservedRunningTime="2026-10-30 22:42:24.057122025 +0000 UTC m=+3594.791485089"
Oct 30 22:42:28.208596 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:42:28.205998    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:42:30.005662 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:30.005629    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:42:30.013331 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:30.013306    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:42:32.497174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:32.491826    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"]
Oct 30 22:42:32.497174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:32.496509    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"
Oct 30 22:42:32.553322 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:32.553284    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"]
Oct 30 22:42:32.598808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:32.598767    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-20ccef\""
Oct 30 22:42:32.598999 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:32.598815    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-20ccef-dockercfg-6cggh\""
Oct 30 22:42:32.653893 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:32.653853    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlt82\" (UniqueName: \"kubernetes.io/projected/25abbe77-bfce-472d-84f4-2d5d9bff0c4d-kube-api-access-dlt82\") pod \"payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx\" (UID: \"25abbe77-bfce-472d-84f4-2d5d9bff0c4d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"
Oct 30 22:42:32.654086 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:32.653998    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25abbe77-bfce-472d-84f4-2d5d9bff0c4d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx\" (UID: \"25abbe77-bfce-472d-84f4-2d5d9bff0c4d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"
Oct 30 22:42:32.755506 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:32.755404    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlt82\" (UniqueName: \"kubernetes.io/projected/25abbe77-bfce-472d-84f4-2d5d9bff0c4d-kube-api-access-dlt82\") pod \"payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx\" (UID: \"25abbe77-bfce-472d-84f4-2d5d9bff0c4d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"
Oct 30 22:42:32.755506 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:32.755488    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25abbe77-bfce-472d-84f4-2d5d9bff0c4d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx\" (UID: \"25abbe77-bfce-472d-84f4-2d5d9bff0c4d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"
Oct 30 22:42:32.756365 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:32.756339    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25abbe77-bfce-472d-84f4-2d5d9bff0c4d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx\" (UID: \"25abbe77-bfce-472d-84f4-2d5d9bff0c4d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"
Oct 30 22:42:32.813987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:32.813959    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlt82\" (UniqueName: \"kubernetes.io/projected/25abbe77-bfce-472d-84f4-2d5d9bff0c4d-kube-api-access-dlt82\") pod \"payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx\" (UID: \"25abbe77-bfce-472d-84f4-2d5d9bff0c4d\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"
Oct 30 22:42:32.816767 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:32.816743    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"
Oct 30 22:42:33.197240 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:33.197206    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"]
Oct 30 22:42:33.198809 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:42:33.198783    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25abbe77_bfce_472d_84f4_2d5d9bff0c4d.slice/crio-e23e88d4bd9e53bf5f8299337980e28039f0f7d6083cf215798f43b639eafeea WatchSource:0}: Error finding container e23e88d4bd9e53bf5f8299337980e28039f0f7d6083cf215798f43b639eafeea: Status 404 returned error can't find the container with id e23e88d4bd9e53bf5f8299337980e28039f0f7d6083cf215798f43b639eafeea
Oct 30 22:42:34.002913 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:34.002874    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx" event={"ID":"25abbe77-bfce-472d-84f4-2d5d9bff0c4d","Type":"ContainerStarted","Data":"beadfa8b8089d06f0ac84a218802270351e6def1a4f37b76f77fc7ee9bce8823"}
Oct 30 22:42:34.003433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:34.002939    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx" event={"ID":"25abbe77-bfce-472d-84f4-2d5d9bff0c4d","Type":"ContainerStarted","Data":"e23e88d4bd9e53bf5f8299337980e28039f0f7d6083cf215798f43b639eafeea"}
Oct 30 22:42:34.003433 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:34.002999    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"
Oct 30 22:42:34.128604 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:34.128548    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx" podStartSLOduration=2.128533182 podStartE2EDuration="2.128533182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:34.125398153 +0000 UTC m=+3604.859761185" watchObservedRunningTime="2026-10-30 22:42:34.128533182 +0000 UTC m=+3604.862896213"
Oct 30 22:42:35.953117 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:35.953087    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"
Oct 30 22:42:45.014697 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:42:45.014667    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"
Oct 30 22:42:58.206668 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:42:58.206627    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:43:12.006809 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:12.006717    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"]
Oct 30 22:43:18.813444 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:18.813406    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"]
Oct 30 22:43:27.037568 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.037526    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c" podUID="fc8615c4-fde1-43e8-b2f0-bae5a77b59b0" containerName="main" containerID="cri-o://66d577cf23c050b395579c305fd55f99b7a2c0f7e50241d8dea958b5e414c2e4" gracePeriod=45
Oct 30 22:43:27.162117 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.162081    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c" podUID="fc8615c4-fde1-43e8-b2f0-bae5a77b59b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.205:8000/ready\": dial tcp 10.134.0.205:8000: connect: connection refused"
Oct 30 22:43:27.306920 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.306855    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:43:27.352290 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.352258    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-model-cache\") pod \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") "
Oct 30 22:43:27.352508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.352340    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-home\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-home\") pod \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") "
Oct 30 22:43:27.352508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.352400    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-kube-api-access-8j84x\" (UniqueName: \"kubernetes.io/projected/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-kube-api-access-8j84x\") pod \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") "
Oct 30 22:43:27.352508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.352427    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-tls-certs\") pod \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") "
Oct 30 22:43:27.352508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.352469    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-dshm\") pod \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") "
Oct 30 22:43:27.352751 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.352519    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-tmp-dir\") pod \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\" (UID: \"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0\") "
Oct 30 22:43:27.352751 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.352583    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-model-cache" pod "fc8615c4-fde1-43e8-b2f0-bae5a77b59b0" (UID: "fc8615c4-fde1-43e8-b2f0-bae5a77b59b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:27.352751 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.352624    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-home" pod "fc8615c4-fde1-43e8-b2f0-bae5a77b59b0" (UID: "fc8615c4-fde1-43e8-b2f0-bae5a77b59b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:27.354189 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.353013    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-tmp-dir" pod "fc8615c4-fde1-43e8-b2f0-bae5a77b59b0" (UID: "fc8615c4-fde1-43e8-b2f0-bae5a77b59b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:27.354189 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.354034    4553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-tmp-dir\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:27.354189 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.354059    4553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-model-cache\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:27.354189 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.354077    4553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-home\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:27.367516 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.367469    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-tls-certs" pod "fc8615c4-fde1-43e8-b2f0-bae5a77b59b0" (UID: "fc8615c4-fde1-43e8-b2f0-bae5a77b59b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:27.367771 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.367751    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-dshm" pod "fc8615c4-fde1-43e8-b2f0-bae5a77b59b0" (UID: "fc8615c4-fde1-43e8-b2f0-bae5a77b59b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:27.369232 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.369047    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-kube-api-access-8j84x" pod "fc8615c4-fde1-43e8-b2f0-bae5a77b59b0" (UID: "fc8615c4-fde1-43e8-b2f0-bae5a77b59b0"). InnerVolumeSpecName "kube-api-access-8j84x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:27.376189 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.376126    4553 generic.go:356] "Generic (PLEG): container finished" podID="fc8615c4-fde1-43e8-b2f0-bae5a77b59b0" containerID="66d577cf23c050b395579c305fd55f99b7a2c0f7e50241d8dea958b5e414c2e4" exitCode=0
Oct 30 22:43:27.376325 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.376217    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"
Oct 30 22:43:27.376325 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.376170    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c" event={"ID":"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0","Type":"ContainerDied","Data":"66d577cf23c050b395579c305fd55f99b7a2c0f7e50241d8dea958b5e414c2e4"}
Oct 30 22:43:27.376325 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.376284    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c" event={"ID":"fc8615c4-fde1-43e8-b2f0-bae5a77b59b0","Type":"ContainerDied","Data":"f503640e99f0a76a5de5e01193e91b2a8a3192dc03536cd68dc2c2537b107ce0"}
Oct 30 22:43:27.376325 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.376305    4553 scope.go:122] "RemoveContainer" containerID="66d577cf23c050b395579c305fd55f99b7a2c0f7e50241d8dea958b5e414c2e4"
Oct 30 22:43:27.389627 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.389606    4553 scope.go:122] "RemoveContainer" containerID="66d577cf23c050b395579c305fd55f99b7a2c0f7e50241d8dea958b5e414c2e4"
Oct 30 22:43:27.389921 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:43:27.389896    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d577cf23c050b395579c305fd55f99b7a2c0f7e50241d8dea958b5e414c2e4\": container with ID starting with 66d577cf23c050b395579c305fd55f99b7a2c0f7e50241d8dea958b5e414c2e4 not found: ID does not exist" containerID="66d577cf23c050b395579c305fd55f99b7a2c0f7e50241d8dea958b5e414c2e4"
Oct 30 22:43:27.390035 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.389933    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d577cf23c050b395579c305fd55f99b7a2c0f7e50241d8dea958b5e414c2e4"} err="failed to get container status \"66d577cf23c050b395579c305fd55f99b7a2c0f7e50241d8dea958b5e414c2e4\": rpc error: code = NotFound desc = could not find container \"66d577cf23c050b395579c305fd55f99b7a2c0f7e50241d8dea958b5e414c2e4\": container with ID starting with 66d577cf23c050b395579c305fd55f99b7a2c0f7e50241d8dea958b5e414c2e4 not found: ID does not exist"
Oct 30 22:43:27.455266 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.455227    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j84x\" (UniqueName: \"kubernetes.io/projected/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-kube-api-access-8j84x\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:27.455266 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.455266    4553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-tls-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:27.455473 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.455281    4553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0-dshm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:27.537843 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.537808    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"]
Oct 30 22:43:27.540606 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.540582    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e90906/auth-test-model-e90906-kserve-7b6c66d68f-rw24c"]
Oct 30 22:43:27.578066 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.577969    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"]
Oct 30 22:43:27.578746 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.578724    4553 cpu_manager.go:414] "containerMap: removing container" podUID="fc8615c4-fde1-43e8-b2f0-bae5a77b59b0" containerName="main"
Oct 30 22:43:27.578858 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.578748    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc8615c4-fde1-43e8-b2f0-bae5a77b59b0" containerName="main"
Oct 30 22:43:27.578997 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.578981    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc8615c4-fde1-43e8-b2f0-bae5a77b59b0" containerName="main"
Oct 30 22:43:27.583986 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.583962    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:27.648865 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.648832    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"]
Oct 30 22:43:27.657904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.657870    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-metrics-certs\") pod \"maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:27.658114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.657997    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:27.658114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.658067    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzmnd\" (UniqueName: \"kubernetes.io/projected/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-kube-api-access-wzmnd\") pod \"maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:27.658281 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.658110    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:27.709128 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.709092    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a6900c-serving-cert\""
Oct 30 22:43:27.759657 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.759618    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-metrics-certs\") pod \"maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:27.759886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.759685    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:27.759886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.759711    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzmnd\" (UniqueName: \"kubernetes.io/projected/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-kube-api-access-wzmnd\") pod \"maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:27.759886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.759851    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:27.760065 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:43:27.759980    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-serving-cert: secret "maas-api-e2e-extmod-a-a6900c-serving-cert" not found
Oct 30 22:43:27.760150 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:43:27.760067    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-maas-api-tls podName:b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62 nodeName:}" failed. No retries permitted until 2026-10-30 22:43:28.260044548 +0000 UTC m=+3658.994407580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-maas-api-tls") pod "maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk" (UID: "b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62") : secret "maas-api-e2e-extmod-a-a6900c-serving-cert" not found
Oct 30 22:43:27.760537 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.760512    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:27.762546 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.762527    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-metrics-certs\") pod \"maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:27.818962 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:27.818925    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzmnd\" (UniqueName: \"kubernetes.io/projected/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-kube-api-access-wzmnd\") pod \"maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:28.202587 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:43:28.202544    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:43:28.266552 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:28.266516    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:28.269217 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:28.269180    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:28.498202 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:28.498099    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:28.876052 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:28.876022    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"]
Oct 30 22:43:28.877221 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:43:28.877190    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d2b8a7_f082_4a3a_adf7_ff2de7b9bb62.slice/crio-4a9578173ad4b365c3d1f9e83536899de7bb08b194a1037d27a9028e5b0a62e4 WatchSource:0}: Error finding container 4a9578173ad4b365c3d1f9e83536899de7bb08b194a1037d27a9028e5b0a62e4: Status 404 returned error can't find the container with id 4a9578173ad4b365c3d1f9e83536899de7bb08b194a1037d27a9028e5b0a62e4
Oct 30 22:43:28.878504 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:28.878486    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:43:29.396181 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:29.396148    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc8615c4-fde1-43e8-b2f0-bae5a77b59b0" path="/var/lib/kubelet/pods/fc8615c4-fde1-43e8-b2f0-bae5a77b59b0/volumes"
Oct 30 22:43:29.396595 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:29.396494    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk" event={"ID":"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62","Type":"ContainerStarted","Data":"4a9578173ad4b365c3d1f9e83536899de7bb08b194a1037d27a9028e5b0a62e4"}
Oct 30 22:43:30.403227 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:30.403180    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk" event={"ID":"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62","Type":"ContainerStarted","Data":"caa2ea07cf39f030724500bfc2bcf128c531f31b4879bf9f4049c5e8ff33bf1d"}
Oct 30 22:43:30.403678 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:30.403326    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:30.520225 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:30.520119    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk" podStartSLOduration=2.568656651 podStartE2EDuration="3.520100734s" totalImagesPullingTime="951.444083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:30.519569302 +0000 UTC m=+3661.253932345" watchObservedRunningTime="2026-10-30 22:43:30.520100734 +0000 UTC m=+3661.254463766"
Oct 30 22:43:31.384148 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:31.384096    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"]
Oct 30 22:43:31.389914 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:31.389887    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"
Oct 30 22:43:31.443584 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:31.443029    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-15e0f6\""
Oct 30 22:43:31.445035 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:31.444999    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"]
Oct 30 22:43:31.491024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:31.490997    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-15e0f6-dockercfg-7vfgc\""
Oct 30 22:43:31.511816 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:31.511783    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e2cb05a-7549-45c9-9eca-db328f6a07cf-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g\" (UID: \"0e2cb05a-7549-45c9-9eca-db328f6a07cf\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"
Oct 30 22:43:31.512077 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:31.512053    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwtcs\" (UniqueName: \"kubernetes.io/projected/0e2cb05a-7549-45c9-9eca-db328f6a07cf-kube-api-access-pwtcs\") pod \"payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g\" (UID: \"0e2cb05a-7549-45c9-9eca-db328f6a07cf\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"
Oct 30 22:43:31.613873 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:31.613841    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e2cb05a-7549-45c9-9eca-db328f6a07cf-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g\" (UID: \"0e2cb05a-7549-45c9-9eca-db328f6a07cf\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"
Oct 30 22:43:31.614104 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:31.613998    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtcs\" (UniqueName: \"kubernetes.io/projected/0e2cb05a-7549-45c9-9eca-db328f6a07cf-kube-api-access-pwtcs\") pod \"payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g\" (UID: \"0e2cb05a-7549-45c9-9eca-db328f6a07cf\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"
Oct 30 22:43:31.614671 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:31.614644    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e2cb05a-7549-45c9-9eca-db328f6a07cf-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g\" (UID: \"0e2cb05a-7549-45c9-9eca-db328f6a07cf\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"
Oct 30 22:43:31.675034 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:31.674938    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtcs\" (UniqueName: \"kubernetes.io/projected/0e2cb05a-7549-45c9-9eca-db328f6a07cf-kube-api-access-pwtcs\") pod \"payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g\" (UID: \"0e2cb05a-7549-45c9-9eca-db328f6a07cf\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"
Oct 30 22:43:31.705888 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:31.705843    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"
Oct 30 22:43:32.092316 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:43:32.092282    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2cb05a_7549_45c9_9eca_db328f6a07cf.slice/crio-8f28b38fd26a6cef02078aa8c5673569a44281c8643b604100e0a6d0296ad6a5 WatchSource:0}: Error finding container 8f28b38fd26a6cef02078aa8c5673569a44281c8643b604100e0a6d0296ad6a5: Status 404 returned error can't find the container with id 8f28b38fd26a6cef02078aa8c5673569a44281c8643b604100e0a6d0296ad6a5
Oct 30 22:43:32.092906 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:32.092825    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"]
Oct 30 22:43:32.419448 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:32.419346    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g" event={"ID":"0e2cb05a-7549-45c9-9eca-db328f6a07cf","Type":"ContainerStarted","Data":"c457452c915348367b417b9ffd760e5506bb588b9a7b1b2db27e654599ada375"}
Oct 30 22:43:32.419448 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:32.419400    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g" event={"ID":"0e2cb05a-7549-45c9-9eca-db328f6a07cf","Type":"ContainerStarted","Data":"8f28b38fd26a6cef02078aa8c5673569a44281c8643b604100e0a6d0296ad6a5"}
Oct 30 22:43:32.419448 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:32.419424    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"
Oct 30 22:43:32.535099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:32.535044    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g" podStartSLOduration=1.5350304769999998 podStartE2EDuration="1.535030477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:32.53463361 +0000 UTC m=+3663.268996643" watchObservedRunningTime="2026-10-30 22:43:32.535030477 +0000 UTC m=+3663.269393505"
Oct 30 22:43:33.835575 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:33.835527    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86" podUID="9ca88c62-838d-4aa1-8c6f-dc1d639eaad2" containerName="main" containerID="cri-o://7a264d367f7fa196d7b63a5fd4049246562503c56b172dffc0c6a23d23f67a4e" gracePeriod=45
Oct 30 22:43:34.102444 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.102375    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:43:34.148681 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.148653    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-model-cache\") pod \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") "
Oct 30 22:43:34.148860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.148740    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-dshm\") pod \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") "
Oct 30 22:43:34.148860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.148777    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-tmp-dir\") pod \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") "
Oct 30 22:43:34.148975 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.148894    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-home\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-home\") pod \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") "
Oct 30 22:43:34.148975 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.148931    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-tls-certs\") pod \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") "
Oct 30 22:43:34.148975 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.148933    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-model-cache" pod "9ca88c62-838d-4aa1-8c6f-dc1d639eaad2" (UID: "9ca88c62-838d-4aa1-8c6f-dc1d639eaad2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:34.149180 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.149002    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-kube-api-access-zgq9s\" (UniqueName: \"kubernetes.io/projected/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-kube-api-access-zgq9s\") pod \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\" (UID: \"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2\") "
Oct 30 22:43:34.149180 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.149125    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-tmp-dir" pod "9ca88c62-838d-4aa1-8c6f-dc1d639eaad2" (UID: "9ca88c62-838d-4aa1-8c6f-dc1d639eaad2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:34.149271 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.149240    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-home" pod "9ca88c62-838d-4aa1-8c6f-dc1d639eaad2" (UID: "9ca88c62-838d-4aa1-8c6f-dc1d639eaad2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:34.149868 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.149841    4553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-home\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:34.150013 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.149872    4553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-model-cache\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:34.150013 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.149892    4553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-tmp-dir\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:34.151170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.151122    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-dshm" pod "9ca88c62-838d-4aa1-8c6f-dc1d639eaad2" (UID: "9ca88c62-838d-4aa1-8c6f-dc1d639eaad2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:34.151294 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.151242    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-tls-certs" pod "9ca88c62-838d-4aa1-8c6f-dc1d639eaad2" (UID: "9ca88c62-838d-4aa1-8c6f-dc1d639eaad2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:34.151294 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.151259    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-kube-api-access-zgq9s" pod "9ca88c62-838d-4aa1-8c6f-dc1d639eaad2" (UID: "9ca88c62-838d-4aa1-8c6f-dc1d639eaad2"). InnerVolumeSpecName "kube-api-access-zgq9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:34.253957 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.253912    4553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-tls-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:34.253957 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.253960    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgq9s\" (UniqueName: \"kubernetes.io/projected/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-kube-api-access-zgq9s\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:34.254247 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.253974    4553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2-dshm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:34.432917 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.432832    4553 generic.go:356] "Generic (PLEG): container finished" podID="9ca88c62-838d-4aa1-8c6f-dc1d639eaad2" containerID="7a264d367f7fa196d7b63a5fd4049246562503c56b172dffc0c6a23d23f67a4e" exitCode=0
Oct 30 22:43:34.432917 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.432880    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"
Oct 30 22:43:34.432917 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.432908    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86" event={"ID":"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2","Type":"ContainerDied","Data":"7a264d367f7fa196d7b63a5fd4049246562503c56b172dffc0c6a23d23f67a4e"}
Oct 30 22:43:34.433216 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.432949    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86" event={"ID":"9ca88c62-838d-4aa1-8c6f-dc1d639eaad2","Type":"ContainerDied","Data":"07a4e869660a8332925088a484469afd406e97b2e38521004de593757e42aa0d"}
Oct 30 22:43:34.433216 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.432965    4553 scope.go:122] "RemoveContainer" containerID="7a264d367f7fa196d7b63a5fd4049246562503c56b172dffc0c6a23d23f67a4e"
Oct 30 22:43:34.444499 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.444470    4553 scope.go:122] "RemoveContainer" containerID="7a264d367f7fa196d7b63a5fd4049246562503c56b172dffc0c6a23d23f67a4e"
Oct 30 22:43:34.444834 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:43:34.444816    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a264d367f7fa196d7b63a5fd4049246562503c56b172dffc0c6a23d23f67a4e\": container with ID starting with 7a264d367f7fa196d7b63a5fd4049246562503c56b172dffc0c6a23d23f67a4e not found: ID does not exist" containerID="7a264d367f7fa196d7b63a5fd4049246562503c56b172dffc0c6a23d23f67a4e"
Oct 30 22:43:34.444914 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.444846    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a264d367f7fa196d7b63a5fd4049246562503c56b172dffc0c6a23d23f67a4e"} err="failed to get container status \"7a264d367f7fa196d7b63a5fd4049246562503c56b172dffc0c6a23d23f67a4e\": rpc error: code = NotFound desc = could not find container \"7a264d367f7fa196d7b63a5fd4049246562503c56b172dffc0c6a23d23f67a4e\": container with ID starting with 7a264d367f7fa196d7b63a5fd4049246562503c56b172dffc0c6a23d23f67a4e not found: ID does not exist"
Oct 30 22:43:34.601780 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.601743    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"]
Oct 30 22:43:34.656308 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:34.656269    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86"]
Oct 30 22:43:35.097588 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:35.097541    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-b1eb3f/auth-test-model-b1eb3f-kserve-7cc88fdb9c-s7v86" podUID="9ca88c62-838d-4aa1-8c6f-dc1d639eaad2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.210:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:43:35.394547 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:35.394445    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca88c62-838d-4aa1-8c6f-dc1d639eaad2" path="/var/lib/kubelet/pods/9ca88c62-838d-4aa1-8c6f-dc1d639eaad2/volumes"
Oct 30 22:43:36.416802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:36.416771    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:43:42.136872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.136829    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"]
Oct 30 22:43:42.137594 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.137575    4553 cpu_manager.go:414] "containerMap: removing container" podUID="9ca88c62-838d-4aa1-8c6f-dc1d639eaad2" containerName="main"
Oct 30 22:43:42.137594 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.137595    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca88c62-838d-4aa1-8c6f-dc1d639eaad2" containerName="main"
Oct 30 22:43:42.137725 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.137715    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca88c62-838d-4aa1-8c6f-dc1d639eaad2" containerName="main"
Oct 30 22:43:42.142832 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.142806    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:42.198657 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.198620    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"]
Oct 30 22:43:42.244719 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.244691    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-20ccef-serving-cert\""
Oct 30 22:43:42.261462 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.261420    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668a3c9f-be16-44ce-91d0-b484530c0d70-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:42.261668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.261477    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkdtv\" (UniqueName: \"kubernetes.io/projected/668a3c9f-be16-44ce-91d0-b484530c0d70-kube-api-access-pkdtv\") pod \"maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:42.261668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.261650    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-maas-api-tls\") pod \"maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:42.261840 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.261818    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-metrics-certs\") pod \"maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:42.363520 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.363481    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-metrics-certs\") pod \"maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:42.363702 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.363605    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668a3c9f-be16-44ce-91d0-b484530c0d70-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:42.363702 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.363653    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkdtv\" (UniqueName: \"kubernetes.io/projected/668a3c9f-be16-44ce-91d0-b484530c0d70-kube-api-access-pkdtv\") pod \"maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:42.363836 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.363717    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-maas-api-tls\") pod \"maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:42.364008 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:43:42.363990    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-serving-cert: secret "maas-api-e2e-extmod-a-20ccef-serving-cert" not found
Oct 30 22:43:42.364085 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:43:42.364071    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-maas-api-tls podName:668a3c9f-be16-44ce-91d0-b484530c0d70 nodeName:}" failed. No retries permitted until 2026-10-30 22:43:42.864048358 +0000 UTC m=+3673.598411384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-maas-api-tls") pod "maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g" (UID: "668a3c9f-be16-44ce-91d0-b484530c0d70") : secret "maas-api-e2e-extmod-a-20ccef-serving-cert" not found
Oct 30 22:43:42.364484 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.364462    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668a3c9f-be16-44ce-91d0-b484530c0d70-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:42.366308 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.366291    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-metrics-certs\") pod \"maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:42.428832 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.428750    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkdtv\" (UniqueName: \"kubernetes.io/projected/668a3c9f-be16-44ce-91d0-b484530c0d70-kube-api-access-pkdtv\") pod \"maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:42.870608 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.870571    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-maas-api-tls\") pod \"maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:42.873329 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:42.873301    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-maas-api-tls\") pod \"maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:43.058531 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:43.058484    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:43.429493 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:43.429409    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"
Oct 30 22:43:43.440978 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:43.440950    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"]
Oct 30 22:43:43.442361 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:43:43.442332    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668a3c9f_be16_44ce_91d0_b484530c0d70.slice/crio-eba92f313ba35adaf0c46be1408699cfe47c5aa92607368e21015bf3f72810ac WatchSource:0}: Error finding container eba92f313ba35adaf0c46be1408699cfe47c5aa92607368e21015bf3f72810ac: Status 404 returned error can't find the container with id eba92f313ba35adaf0c46be1408699cfe47c5aa92607368e21015bf3f72810ac
Oct 30 22:43:43.494792 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:43.494757    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g" event={"ID":"668a3c9f-be16-44ce-91d0-b484530c0d70","Type":"ContainerStarted","Data":"eba92f313ba35adaf0c46be1408699cfe47c5aa92607368e21015bf3f72810ac"}
Oct 30 22:43:44.502732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:44.502693    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g" event={"ID":"668a3c9f-be16-44ce-91d0-b484530c0d70","Type":"ContainerStarted","Data":"56b0b37317cef562f8f72f8b732f55ba2cd42738e60219e2102456f65f25e570"}
Oct 30 22:43:44.503263 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:44.502810    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:44.624179 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:44.624095    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g" podStartSLOduration=1.789004873 podStartE2EDuration="2.624081418s" totalImagesPullingTime="835.076545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:44.623182076 +0000 UTC m=+3675.357545109" watchObservedRunningTime="2026-10-30 22:43:44.624081418 +0000 UTC m=+3675.358444448"
Oct 30 22:43:50.515899 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:43:50.515868    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:43:58.202127 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:43:58.202083    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:44:16.268968 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:16.268931    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"]
Oct 30 22:44:28.204260 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:44:28.204217    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:44:29.304713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:29.304679    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"]
Oct 30 22:44:29.310981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:29.310954    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"
Oct 30 22:44:29.366125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:29.366090    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"]
Oct 30 22:44:29.411466 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:29.411439    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-55edb1\""
Oct 30 22:44:29.412568 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:29.412554    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-55edb1-dockercfg-w4rl4\""
Oct 30 22:44:29.488184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:29.488128    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51090a1f-1c13-4daa-b493-1cb54373cb43-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f\" (UID: \"51090a1f-1c13-4daa-b493-1cb54373cb43\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"
Oct 30 22:44:29.488401 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:29.488378    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlgnc\" (UniqueName: \"kubernetes.io/projected/51090a1f-1c13-4daa-b493-1cb54373cb43-kube-api-access-vlgnc\") pod \"payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f\" (UID: \"51090a1f-1c13-4daa-b493-1cb54373cb43\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"
Oct 30 22:44:29.590017 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:29.589979    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlgnc\" (UniqueName: \"kubernetes.io/projected/51090a1f-1c13-4daa-b493-1cb54373cb43-kube-api-access-vlgnc\") pod \"payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f\" (UID: \"51090a1f-1c13-4daa-b493-1cb54373cb43\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"
Oct 30 22:44:29.590247 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:29.590213    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51090a1f-1c13-4daa-b493-1cb54373cb43-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f\" (UID: \"51090a1f-1c13-4daa-b493-1cb54373cb43\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"
Oct 30 22:44:29.590930 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:29.590902    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51090a1f-1c13-4daa-b493-1cb54373cb43-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f\" (UID: \"51090a1f-1c13-4daa-b493-1cb54373cb43\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"
Oct 30 22:44:29.649041 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:29.649006    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlgnc\" (UniqueName: \"kubernetes.io/projected/51090a1f-1c13-4daa-b493-1cb54373cb43-kube-api-access-vlgnc\") pod \"payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f\" (UID: \"51090a1f-1c13-4daa-b493-1cb54373cb43\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"
Oct 30 22:44:29.925632 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:29.925532    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"
Oct 30 22:44:30.309228 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:30.309196    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"]
Oct 30 22:44:30.309691 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:44:30.309465    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51090a1f_1c13_4daa_b493_1cb54373cb43.slice/crio-ee54ff90832d1478328dc5b37bbb4e93a9d5f67759ab6acdf06fc91ba71e72c7 WatchSource:0}: Error finding container ee54ff90832d1478328dc5b37bbb4e93a9d5f67759ab6acdf06fc91ba71e72c7: Status 404 returned error can't find the container with id ee54ff90832d1478328dc5b37bbb4e93a9d5f67759ab6acdf06fc91ba71e72c7
Oct 30 22:44:30.823726 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:30.823669    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f" event={"ID":"51090a1f-1c13-4daa-b493-1cb54373cb43","Type":"ContainerStarted","Data":"52e361dfba7742579f30a1d7967176043800d6b9095de6e35bd9423409ec46c4"}
Oct 30 22:44:30.823932 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:30.823774    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f" event={"ID":"51090a1f-1c13-4daa-b493-1cb54373cb43","Type":"ContainerStarted","Data":"ee54ff90832d1478328dc5b37bbb4e93a9d5f67759ab6acdf06fc91ba71e72c7"}
Oct 30 22:44:30.823932 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:30.823804    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"
Oct 30 22:44:30.942122 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:30.942067    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f" podStartSLOduration=1.942052404 podStartE2EDuration="1.942052404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:30.939220702 +0000 UTC m=+3721.673583729" watchObservedRunningTime="2026-10-30 22:44:30.942052404 +0000 UTC m=+3721.676415472"
Oct 30 22:44:31.291022 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.290974    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw" podUID="b996db18-28c1-47da-889e-b99c118f1c42" containerName="main" containerID="cri-o://941ee63af6ba3f69b74ace21752fadb2d452dbf0cf95dec5f5ebc7c1686f20ce" gracePeriod=45
Oct 30 22:44:31.583042 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.583016    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:44:31.717689 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.717652    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b996db18-28c1-47da-889e-b99c118f1c42-kube-api-access-gdf5m\" (UniqueName: \"kubernetes.io/projected/b996db18-28c1-47da-889e-b99c118f1c42-kube-api-access-gdf5m\") pod \"b996db18-28c1-47da-889e-b99c118f1c42\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") "
Oct 30 22:44:31.717909 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.717788    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-home\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-home\") pod \"b996db18-28c1-47da-889e-b99c118f1c42\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") "
Oct 30 22:44:31.717909 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.717879    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-model-cache\") pod \"b996db18-28c1-47da-889e-b99c118f1c42\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") "
Oct 30 22:44:31.718034 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.717913    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-dshm\") pod \"b996db18-28c1-47da-889e-b99c118f1c42\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") "
Oct 30 22:44:31.718091 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.718034    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-home" pod "b996db18-28c1-47da-889e-b99c118f1c42" (UID: "b996db18-28c1-47da-889e-b99c118f1c42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:31.718091 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.718070    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b996db18-28c1-47da-889e-b99c118f1c42-tls-certs\" (UniqueName: \"kubernetes.io/secret/b996db18-28c1-47da-889e-b99c118f1c42-tls-certs\") pod \"b996db18-28c1-47da-889e-b99c118f1c42\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") "
Oct 30 22:44:31.718229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.718162    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-tmp-dir\") pod \"b996db18-28c1-47da-889e-b99c118f1c42\" (UID: \"b996db18-28c1-47da-889e-b99c118f1c42\") "
Oct 30 22:44:31.718229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.718194    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-model-cache" pod "b996db18-28c1-47da-889e-b99c118f1c42" (UID: "b996db18-28c1-47da-889e-b99c118f1c42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:31.718491 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.718472    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-tmp-dir" pod "b996db18-28c1-47da-889e-b99c118f1c42" (UID: "b996db18-28c1-47da-889e-b99c118f1c42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:31.718684 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.718668    4553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-model-cache\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:31.718684 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.718687    4553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-tmp-dir\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:31.718788 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.718696    4553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-home\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:31.720435 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.720411    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b996db18-28c1-47da-889e-b99c118f1c42-kube-api-access-gdf5m" pod "b996db18-28c1-47da-889e-b99c118f1c42" (UID: "b996db18-28c1-47da-889e-b99c118f1c42"). InnerVolumeSpecName "kube-api-access-gdf5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:31.720908 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.720888    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-dshm" pod "b996db18-28c1-47da-889e-b99c118f1c42" (UID: "b996db18-28c1-47da-889e-b99c118f1c42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:31.720908 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.720891    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b996db18-28c1-47da-889e-b99c118f1c42-tls-certs" pod "b996db18-28c1-47da-889e-b99c118f1c42" (UID: "b996db18-28c1-47da-889e-b99c118f1c42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:31.819970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.819880    4553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b996db18-28c1-47da-889e-b99c118f1c42-dshm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:31.819970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.819917    4553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b996db18-28c1-47da-889e-b99c118f1c42-tls-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:31.819970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.819932    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdf5m\" (UniqueName: \"kubernetes.io/projected/b996db18-28c1-47da-889e-b99c118f1c42-kube-api-access-gdf5m\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:31.832105 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.832073    4553 generic.go:356] "Generic (PLEG): container finished" podID="b996db18-28c1-47da-889e-b99c118f1c42" containerID="941ee63af6ba3f69b74ace21752fadb2d452dbf0cf95dec5f5ebc7c1686f20ce" exitCode=0
Oct 30 22:44:31.832315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.832163    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"
Oct 30 22:44:31.832315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.832167    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw" event={"ID":"b996db18-28c1-47da-889e-b99c118f1c42","Type":"ContainerDied","Data":"941ee63af6ba3f69b74ace21752fadb2d452dbf0cf95dec5f5ebc7c1686f20ce"}
Oct 30 22:44:31.832315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.832214    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw" event={"ID":"b996db18-28c1-47da-889e-b99c118f1c42","Type":"ContainerDied","Data":"fa81857d1c163cfff810925ce7356ef4577e6b19eb99bec512317115b5a914d5"}
Oct 30 22:44:31.832315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.832231    4553 scope.go:122] "RemoveContainer" containerID="941ee63af6ba3f69b74ace21752fadb2d452dbf0cf95dec5f5ebc7c1686f20ce"
Oct 30 22:44:31.844687 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.844661    4553 scope.go:122] "RemoveContainer" containerID="941ee63af6ba3f69b74ace21752fadb2d452dbf0cf95dec5f5ebc7c1686f20ce"
Oct 30 22:44:31.845016 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:44:31.844991    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941ee63af6ba3f69b74ace21752fadb2d452dbf0cf95dec5f5ebc7c1686f20ce\": container with ID starting with 941ee63af6ba3f69b74ace21752fadb2d452dbf0cf95dec5f5ebc7c1686f20ce not found: ID does not exist" containerID="941ee63af6ba3f69b74ace21752fadb2d452dbf0cf95dec5f5ebc7c1686f20ce"
Oct 30 22:44:31.845067 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:31.845029    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941ee63af6ba3f69b74ace21752fadb2d452dbf0cf95dec5f5ebc7c1686f20ce"} err="failed to get container status \"941ee63af6ba3f69b74ace21752fadb2d452dbf0cf95dec5f5ebc7c1686f20ce\": rpc error: code = NotFound desc = could not find container \"941ee63af6ba3f69b74ace21752fadb2d452dbf0cf95dec5f5ebc7c1686f20ce\": container with ID starting with 941ee63af6ba3f69b74ace21752fadb2d452dbf0cf95dec5f5ebc7c1686f20ce not found: ID does not exist"
Oct 30 22:44:32.003277 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:32.003240    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"]
Oct 30 22:44:32.009937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:32.009906    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b4deef/auth-test-model-b4deef-kserve-777854dc7-wpmlw"]
Oct 30 22:44:33.394637 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:33.394601    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b996db18-28c1-47da-889e-b99c118f1c42" path="/var/lib/kubelet/pods/b996db18-28c1-47da-889e-b99c118f1c42/volumes"
Oct 30 22:44:41.836530 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:44:41.836501    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"
Oct 30 22:44:58.202987 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:44:58.202948    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:45:00.236806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:00.236773    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"]
Oct 30 22:45:00.237731 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:00.237708    4553 cpu_manager.go:414] "containerMap: removing container" podUID="b996db18-28c1-47da-889e-b99c118f1c42" containerName="main"
Oct 30 22:45:00.237848 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:00.237733    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b996db18-28c1-47da-889e-b99c118f1c42" containerName="main"
Oct 30 22:45:00.237959 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:00.237945    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="b996db18-28c1-47da-889e-b99c118f1c42" containerName="main"
Oct 30 22:45:00.242567 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:00.242543    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"
Oct 30 22:45:00.293814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:00.293773    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cfhp\" (UniqueName: \"kubernetes.io/projected/7ddaa5a7-2fea-48d3-8698-1d4d4b222a65-kube-api-access-6cfhp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f\" (UID: \"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"
Oct 30 22:45:00.294028 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:00.293866    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddaa5a7-2fea-48d3-8698-1d4d4b222a65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f\" (UID: \"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"
Oct 30 22:45:00.325581 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:00.325483    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"]
Oct 30 22:45:00.346406 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:00.346377    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gjgvl\""
Oct 30 22:45:00.396082 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:00.396033    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cfhp\" (UniqueName: \"kubernetes.io/projected/7ddaa5a7-2fea-48d3-8698-1d4d4b222a65-kube-api-access-6cfhp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f\" (UID: \"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"
Oct 30 22:45:00.396311 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:00.396154    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddaa5a7-2fea-48d3-8698-1d4d4b222a65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f\" (UID: \"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"
Oct 30 22:45:00.397235 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:00.397207    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddaa5a7-2fea-48d3-8698-1d4d4b222a65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f\" (UID: \"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"
Oct 30 22:45:00.455637 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:00.455605    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cfhp\" (UniqueName: \"kubernetes.io/projected/7ddaa5a7-2fea-48d3-8698-1d4d4b222a65-kube-api-access-6cfhp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f\" (UID: \"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"
Oct 30 22:45:00.557736 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:00.557647    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"
Oct 30 22:45:00.941936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:00.941906    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"]
Oct 30 22:45:00.943212 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:45:00.943186    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddaa5a7_2fea_48d3_8698_1d4d4b222a65.slice/crio-3a9e1b41c72887bb9361a2f4ebcdc262f49468734d67ac841754ff67959979dc WatchSource:0}: Error finding container 3a9e1b41c72887bb9361a2f4ebcdc262f49468734d67ac841754ff67959979dc: Status 404 returned error can't find the container with id 3a9e1b41c72887bb9361a2f4ebcdc262f49468734d67ac841754ff67959979dc
Oct 30 22:45:01.032518 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:01.032477    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f" event={"ID":"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65","Type":"ContainerStarted","Data":"3a9e1b41c72887bb9361a2f4ebcdc262f49468734d67ac841754ff67959979dc"}
Oct 30 22:45:02.040810 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:02.040717    4553 generic.go:356] "Generic (PLEG): container finished" podID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65" containerID="48e6c07e5693d791b7536445319cfe7d0e7f6c44e92b439f2af753f96f70060b" exitCode=60
Oct 30 22:45:02.040810 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:02.040795    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f" event={"ID":"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65","Type":"ContainerDied","Data":"48e6c07e5693d791b7536445319cfe7d0e7f6c44e92b439f2af753f96f70060b"}
Oct 30 22:45:02.041292 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:02.041182    4553 scope.go:122] "RemoveContainer" containerID="48e6c07e5693d791b7536445319cfe7d0e7f6c44e92b439f2af753f96f70060b"
Oct 30 22:45:03.050905 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:03.050866    4553 generic.go:356] "Generic (PLEG): container finished" podID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65" containerID="1d6059ebd1a9201b9053d425fda9a3bcd4a634d9f58383ecf4e51dda059cf996" exitCode=60
Oct 30 22:45:03.051410 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:03.050923    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f" event={"ID":"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65","Type":"ContainerDied","Data":"1d6059ebd1a9201b9053d425fda9a3bcd4a634d9f58383ecf4e51dda059cf996"}
Oct 30 22:45:03.051410 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:03.050983    4553 scope.go:122] "RemoveContainer" containerID="48e6c07e5693d791b7536445319cfe7d0e7f6c44e92b439f2af753f96f70060b"
Oct 30 22:45:03.051410 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:03.051346    4553 scope.go:122] "RemoveContainer" containerID="1d6059ebd1a9201b9053d425fda9a3bcd4a634d9f58383ecf4e51dda059cf996"
Oct 30 22:45:03.051972 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:45:03.051944    4553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f_odh-ai-gateway-infra(7ddaa5a7-2fea-48d3-8698-1d4d4b222a65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f" podUID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65"
Oct 30 22:45:04.060006 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:04.059980    4553 scope.go:122] "RemoveContainer" containerID="1d6059ebd1a9201b9053d425fda9a3bcd4a634d9f58383ecf4e51dda059cf996"
Oct 30 22:45:04.060493 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:45:04.060396    4553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f_odh-ai-gateway-infra(7ddaa5a7-2fea-48d3-8698-1d4d4b222a65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f" podUID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65"
Oct 30 22:45:15.386200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:15.386170    4553 scope.go:122] "RemoveContainer" containerID="1d6059ebd1a9201b9053d425fda9a3bcd4a634d9f58383ecf4e51dda059cf996"
Oct 30 22:45:16.152523 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:16.152484    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f" event={"ID":"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65","Type":"ContainerStarted","Data":"35152d15163748b83f5076c4379a1860557164e2f3454e0bb9eb17fd912956c4"}
Oct 30 22:45:16.268987 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:16.268925    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f" podStartSLOduration=16.268910125 podStartE2EDuration="16.268910125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:16.267202978 +0000 UTC m=+3767.001566010" watchObservedRunningTime="2026-10-30 22:45:16.268910125 +0000 UTC m=+3767.003273156"
Oct 30 22:45:17.162628 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:17.162590    4553 generic.go:356] "Generic (PLEG): container finished" podID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65" containerID="35152d15163748b83f5076c4379a1860557164e2f3454e0bb9eb17fd912956c4" exitCode=60
Oct 30 22:45:17.163125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:17.162659    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f" event={"ID":"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65","Type":"ContainerDied","Data":"35152d15163748b83f5076c4379a1860557164e2f3454e0bb9eb17fd912956c4"}
Oct 30 22:45:17.163125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:17.162739    4553 scope.go:122] "RemoveContainer" containerID="1d6059ebd1a9201b9053d425fda9a3bcd4a634d9f58383ecf4e51dda059cf996"
Oct 30 22:45:17.163320 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:17.163233    4553 scope.go:122] "RemoveContainer" containerID="35152d15163748b83f5076c4379a1860557164e2f3454e0bb9eb17fd912956c4"
Oct 30 22:45:17.163650 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:45:17.163624    4553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f_odh-ai-gateway-infra(7ddaa5a7-2fea-48d3-8698-1d4d4b222a65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f" podUID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65"
Oct 30 22:45:17.279835 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:17.279799    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"]
Oct 30 22:45:17.889400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:17.889341    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"]
Oct 30 22:45:17.895530 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:17.895505    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"
Oct 30 22:45:17.949602 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:17.949566    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"]
Oct 30 22:45:17.950079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:17.949931    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-69058c\""
Oct 30 22:45:17.950244 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:17.950179    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-69058c-dockercfg-bmqdp\""
Oct 30 22:45:18.019821 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.019781    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"]
Oct 30 22:45:18.024358 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.024326    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"
Oct 30 22:45:18.034180 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.034125    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv\" (UID: \"1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"
Oct 30 22:45:18.035061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.035032    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljkk\" (UniqueName: \"kubernetes.io/projected/1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b-kube-api-access-sljkk\") pod \"payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv\" (UID: \"1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"
Oct 30 22:45:18.079672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.079635    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"]
Oct 30 22:45:18.136717 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.136682    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppc8t\" (UniqueName: \"kubernetes.io/projected/74a81c04-db18-427e-b73d-f4bc5ea33774-kube-api-access-ppc8t\") pod \"payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s\" (UID: \"74a81c04-db18-427e-b73d-f4bc5ea33774\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"
Oct 30 22:45:18.136906 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.136727    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljkk\" (UniqueName: \"kubernetes.io/projected/1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b-kube-api-access-sljkk\") pod \"payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv\" (UID: \"1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"
Oct 30 22:45:18.136947 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.136899    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a81c04-db18-427e-b73d-f4bc5ea33774-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s\" (UID: \"74a81c04-db18-427e-b73d-f4bc5ea33774\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"
Oct 30 22:45:18.136984 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.136972    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv\" (UID: \"1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"
Oct 30 22:45:18.137756 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.137708    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv\" (UID: \"1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"
Oct 30 22:45:18.196317 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.196281    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljkk\" (UniqueName: \"kubernetes.io/projected/1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b-kube-api-access-sljkk\") pod \"payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv\" (UID: \"1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"
Oct 30 22:45:18.211481 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.211440    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"
Oct 30 22:45:18.238256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.238221    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a81c04-db18-427e-b73d-f4bc5ea33774-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s\" (UID: \"74a81c04-db18-427e-b73d-f4bc5ea33774\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"
Oct 30 22:45:18.238448 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.238433    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppc8t\" (UniqueName: \"kubernetes.io/projected/74a81c04-db18-427e-b73d-f4bc5ea33774-kube-api-access-ppc8t\") pod \"payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s\" (UID: \"74a81c04-db18-427e-b73d-f4bc5ea33774\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"
Oct 30 22:45:18.239273 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.239247    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a81c04-db18-427e-b73d-f4bc5ea33774-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s\" (UID: \"74a81c04-db18-427e-b73d-f4bc5ea33774\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"
Oct 30 22:45:18.308050 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.306780    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppc8t\" (UniqueName: \"kubernetes.io/projected/74a81c04-db18-427e-b73d-f4bc5ea33774-kube-api-access-ppc8t\") pod \"payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s\" (UID: \"74a81c04-db18-427e-b73d-f4bc5ea33774\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"
Oct 30 22:45:18.343692 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.343611    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"
Oct 30 22:45:18.362283 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.362256    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"
Oct 30 22:45:18.543764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.543681    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ddaa5a7-2fea-48d3-8698-1d4d4b222a65-kube-api-access-6cfhp\" (UniqueName: \"kubernetes.io/projected/7ddaa5a7-2fea-48d3-8698-1d4d4b222a65-kube-api-access-6cfhp\") pod \"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65\" (UID: \"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65\") "
Oct 30 22:45:18.543945 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.543931    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ddaa5a7-2fea-48d3-8698-1d4d4b222a65-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddaa5a7-2fea-48d3-8698-1d4d4b222a65-maas-api-service-ca\") pod \"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65\" (UID: \"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65\") "
Oct 30 22:45:18.544298 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.544267    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ddaa5a7-2fea-48d3-8698-1d4d4b222a65-maas-api-service-ca" pod "7ddaa5a7-2fea-48d3-8698-1d4d4b222a65" (UID: "7ddaa5a7-2fea-48d3-8698-1d4d4b222a65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.544442 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.544427    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddaa5a7-2fea-48d3-8698-1d4d4b222a65-maas-api-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.546114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.546092    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ddaa5a7-2fea-48d3-8698-1d4d4b222a65-kube-api-access-6cfhp" pod "7ddaa5a7-2fea-48d3-8698-1d4d4b222a65" (UID: "7ddaa5a7-2fea-48d3-8698-1d4d4b222a65"). InnerVolumeSpecName "kube-api-access-6cfhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:18.606599 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.606566    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"]
Oct 30 22:45:18.607476 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:45:18.607444    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aaa183a_c1f4_4b5f_aecb_5f11e4575c1b.slice/crio-b9fcaa2dd349105979806952918f712f5e3e0baaad913052ddc0e8c9f30c49d1 WatchSource:0}: Error finding container b9fcaa2dd349105979806952918f712f5e3e0baaad913052ddc0e8c9f30c49d1: Status 404 returned error can't find the container with id b9fcaa2dd349105979806952918f712f5e3e0baaad913052ddc0e8c9f30c49d1
Oct 30 22:45:18.646032 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.645996    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cfhp\" (UniqueName: \"kubernetes.io/projected/7ddaa5a7-2fea-48d3-8698-1d4d4b222a65-kube-api-access-6cfhp\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.731314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:18.731283    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"]
Oct 30 22:45:18.732373 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:45:18.732344    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a81c04_db18_427e_b73d_f4bc5ea33774.slice/crio-8fd4a7404e0cfc8b99b234aa6d818d62dcfdc2040f98f14e1ff5e1975d62ce04 WatchSource:0}: Error finding container 8fd4a7404e0cfc8b99b234aa6d818d62dcfdc2040f98f14e1ff5e1975d62ce04: Status 404 returned error can't find the container with id 8fd4a7404e0cfc8b99b234aa6d818d62dcfdc2040f98f14e1ff5e1975d62ce04
Oct 30 22:45:19.180929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:19.180888    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s" event={"ID":"74a81c04-db18-427e-b73d-f4bc5ea33774","Type":"ContainerStarted","Data":"3fb33e986ee6022c27a5506097e57ea43c4a7a031a316cd747e74546f12da409"}
Oct 30 22:45:19.181189 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:19.180951    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s" event={"ID":"74a81c04-db18-427e-b73d-f4bc5ea33774","Type":"ContainerStarted","Data":"8fd4a7404e0cfc8b99b234aa6d818d62dcfdc2040f98f14e1ff5e1975d62ce04"}
Oct 30 22:45:19.181189 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:19.181114    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"
Oct 30 22:45:19.182731 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:19.182706    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv" event={"ID":"1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b","Type":"ContainerStarted","Data":"6d3c0690a47b1cf0d8d04c5a0df9753145a3ef87d83108e04fd03f3448529a9b"}
Oct 30 22:45:19.182862 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:19.182754    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv" event={"ID":"1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b","Type":"ContainerStarted","Data":"b9fcaa2dd349105979806952918f712f5e3e0baaad913052ddc0e8c9f30c49d1"}
Oct 30 22:45:19.182862 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:19.182779    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"
Oct 30 22:45:19.184181 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:19.184128    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"
Oct 30 22:45:19.184181 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:19.184162    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f" event={"ID":"7ddaa5a7-2fea-48d3-8698-1d4d4b222a65","Type":"ContainerDied","Data":"3a9e1b41c72887bb9361a2f4ebcdc262f49468734d67ac841754ff67959979dc"}
Oct 30 22:45:19.184372 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:19.184207    4553 scope.go:122] "RemoveContainer" containerID="35152d15163748b83f5076c4379a1860557164e2f3454e0bb9eb17fd912956c4"
Oct 30 22:45:19.298476 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:19.298418    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s" podStartSLOduration=2.298391602 podStartE2EDuration="2.298391602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:19.295888123 +0000 UTC m=+3770.030251191" watchObservedRunningTime="2026-10-30 22:45:19.298391602 +0000 UTC m=+3770.032754633"
Oct 30 22:45:19.416035 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:19.415956    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"]
Oct 30 22:45:19.419053 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:19.418999    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0e450a-29890005-c5m5f"]
Oct 30 22:45:19.533390 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:19.533272    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv" podStartSLOduration=2.533257202 podStartE2EDuration="2.533257202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:19.530163389 +0000 UTC m=+3770.264526420" watchObservedRunningTime="2026-10-30 22:45:19.533257202 +0000 UTC m=+3770.267620233"
Oct 30 22:45:21.394955 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:21.394916    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65" path="/var/lib/kubelet/pods/7ddaa5a7-2fea-48d3-8698-1d4d4b222a65/volumes"
Oct 30 22:45:24.852801 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:24.852770    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"]
Oct 30 22:45:28.202859 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:45:28.202826    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:45:30.195178 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:30.195121    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"
Oct 30 22:45:30.195659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:30.195223    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"
Oct 30 22:45:38.930473 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:38.930437    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"]
Oct 30 22:45:38.931404 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:38.931384    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65" containerName="cleanup"
Oct 30 22:45:38.931404 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:38.931405    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65" containerName="cleanup"
Oct 30 22:45:38.931536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:38.931417    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65" containerName="cleanup"
Oct 30 22:45:38.931536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:38.931423    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65" containerName="cleanup"
Oct 30 22:45:38.931536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:38.931443    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65" containerName="cleanup"
Oct 30 22:45:38.931536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:38.931448    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65" containerName="cleanup"
Oct 30 22:45:38.931536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:38.931537    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65" containerName="cleanup"
Oct 30 22:45:38.931701 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:38.931553    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65" containerName="cleanup"
Oct 30 22:45:38.935259 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:38.935243    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:38.993342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:38.993312    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-32af64-openshift-default-dockercfg-lxphz\""
Oct 30 22:45:39.022298 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.022247    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-credential-socket\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.022656 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.022634    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-envoy\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.022884 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.022869    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-data\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.023036 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.023021    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-workload-socket\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.023221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.023204    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-workload-certs\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.023375 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.023359    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-podinfo\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.023541 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.023521    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncpgb\" (UniqueName: \"kubernetes.io/projected/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-kube-api-access-ncpgb\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.023625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.023574    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istiod-ca-cert\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.023686 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.023665    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-token\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.049270 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.049238    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"]
Oct 30 22:45:39.125507 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.125468    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-data\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.125507 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.125506    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-workload-socket\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.125797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.125539    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-workload-certs\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.125797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.125560    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-podinfo\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.125797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.125587    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncpgb\" (UniqueName: \"kubernetes.io/projected/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-kube-api-access-ncpgb\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.125797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.125618    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istiod-ca-cert\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.125797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.125678    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-token\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.126088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.125810    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-credential-socket\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.126088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.125854    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-envoy\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.126088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.125947    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-data\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.126088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.125999    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-workload-socket\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.126088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.126046    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-workload-certs\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.126487 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.126248    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-credential-socket\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.126487 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.126435    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istiod-ca-cert\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.128222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.128201    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-envoy\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.128462 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.128446    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-podinfo\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.189795 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.189713    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-token\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.189795 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.189776    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncpgb\" (UniqueName: \"kubernetes.io/projected/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-kube-api-access-ncpgb\") pod \"e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") " pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.249848 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.249815    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:39.681972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.681945    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"]
Oct 30 22:45:39.683392 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:45:39.683361    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f83e3e_757f_495d_9734_e3f7a77d9b9d.slice/crio-0181e7aec2089794c965d3c1bc41bf8219fcb219695487f609659f6d75c1288c WatchSource:0}: Error finding container 0181e7aec2089794c965d3c1bc41bf8219fcb219695487f609659f6d75c1288c: Status 404 returned error can't find the container with id 0181e7aec2089794c965d3c1bc41bf8219fcb219695487f609659f6d75c1288c
Oct 30 22:45:39.686296 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.686264    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:45:39.686399 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.686331    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:45:39.686399 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.686360    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:45:39.741567 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.741537    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"]
Oct 30 22:45:39.878163 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:39.876611    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9" podUID="0f3f457a-83fb-4bcd-a884-d9a988379046" containerName="main" containerID="cri-o://2329eccb1d9a8a74e2078baa52a15fc46a64f3648c5f3847dc6f4cf16d935808" gracePeriod=45
Oct 30 22:45:40.171345 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.171317    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:45:40.344981 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.344941    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" event={"ID":"d5f83e3e-757f-495d-9734-e3f7a77d9b9d","Type":"ContainerStarted","Data":"3e931b688cc5dd3cde0f474d1373e89336409217d59519d2bdd4ecdcf49ffd1d"}
Oct 30 22:45:40.345203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.345011    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" event={"ID":"d5f83e3e-757f-495d-9734-e3f7a77d9b9d","Type":"ContainerStarted","Data":"0181e7aec2089794c965d3c1bc41bf8219fcb219695487f609659f6d75c1288c"}
Oct 30 22:45:40.346609 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.346582    4553 generic.go:356] "Generic (PLEG): container finished" podID="0f3f457a-83fb-4bcd-a884-d9a988379046" containerID="2329eccb1d9a8a74e2078baa52a15fc46a64f3648c5f3847dc6f4cf16d935808" exitCode=0
Oct 30 22:45:40.346712 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.346637    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"
Oct 30 22:45:40.346712 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.346652    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9" event={"ID":"0f3f457a-83fb-4bcd-a884-d9a988379046","Type":"ContainerDied","Data":"2329eccb1d9a8a74e2078baa52a15fc46a64f3648c5f3847dc6f4cf16d935808"}
Oct 30 22:45:40.346712 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.346683    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9" event={"ID":"0f3f457a-83fb-4bcd-a884-d9a988379046","Type":"ContainerDied","Data":"27289f1632468512804285ce64c10f58421322091e674a3079968a0964c0a3aa"}
Oct 30 22:45:40.346712 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.346700    4553 scope.go:122] "RemoveContainer" containerID="2329eccb1d9a8a74e2078baa52a15fc46a64f3648c5f3847dc6f4cf16d935808"
Oct 30 22:45:40.351794 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.351731    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3f457a-83fb-4bcd-a884-d9a988379046-kube-api-access-xt67j\" (UniqueName: \"kubernetes.io/projected/0f3f457a-83fb-4bcd-a884-d9a988379046-kube-api-access-xt67j\") pod \"0f3f457a-83fb-4bcd-a884-d9a988379046\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") "
Oct 30 22:45:40.351794 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.351779    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f3f457a-83fb-4bcd-a884-d9a988379046-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3f457a-83fb-4bcd-a884-d9a988379046-tls-certs\") pod \"0f3f457a-83fb-4bcd-a884-d9a988379046\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") "
Oct 30 22:45:40.351939 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.351895    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-model-cache\") pod \"0f3f457a-83fb-4bcd-a884-d9a988379046\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") "
Oct 30 22:45:40.351986 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.351979    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-dshm\") pod \"0f3f457a-83fb-4bcd-a884-d9a988379046\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") "
Oct 30 22:45:40.352031 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.352007    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-tmp-dir\") pod \"0f3f457a-83fb-4bcd-a884-d9a988379046\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") "
Oct 30 22:45:40.352071 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.352027    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-home\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-home\") pod \"0f3f457a-83fb-4bcd-a884-d9a988379046\" (UID: \"0f3f457a-83fb-4bcd-a884-d9a988379046\") "
Oct 30 22:45:40.352590 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.352401    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-model-cache" pod "0f3f457a-83fb-4bcd-a884-d9a988379046" (UID: "0f3f457a-83fb-4bcd-a884-d9a988379046"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:40.352590 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.352521    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-home" pod "0f3f457a-83fb-4bcd-a884-d9a988379046" (UID: "0f3f457a-83fb-4bcd-a884-d9a988379046"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:40.352590 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.352533    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-tmp-dir" pod "0f3f457a-83fb-4bcd-a884-d9a988379046" (UID: "0f3f457a-83fb-4bcd-a884-d9a988379046"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:40.352819 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.352607    4553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-model-cache\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:40.354600 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.354567    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-dshm" pod "0f3f457a-83fb-4bcd-a884-d9a988379046" (UID: "0f3f457a-83fb-4bcd-a884-d9a988379046"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:40.354690 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.354603    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3f457a-83fb-4bcd-a884-d9a988379046-kube-api-access-xt67j" pod "0f3f457a-83fb-4bcd-a884-d9a988379046" (UID: "0f3f457a-83fb-4bcd-a884-d9a988379046"). InnerVolumeSpecName "kube-api-access-xt67j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:40.354690 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.354663    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f3f457a-83fb-4bcd-a884-d9a988379046-tls-certs" pod "0f3f457a-83fb-4bcd-a884-d9a988379046" (UID: "0f3f457a-83fb-4bcd-a884-d9a988379046"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:40.358218 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.358195    4553 scope.go:122] "RemoveContainer" containerID="2329eccb1d9a8a74e2078baa52a15fc46a64f3648c5f3847dc6f4cf16d935808"
Oct 30 22:45:40.358518 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:45:40.358497    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2329eccb1d9a8a74e2078baa52a15fc46a64f3648c5f3847dc6f4cf16d935808\": container with ID starting with 2329eccb1d9a8a74e2078baa52a15fc46a64f3648c5f3847dc6f4cf16d935808 not found: ID does not exist" containerID="2329eccb1d9a8a74e2078baa52a15fc46a64f3648c5f3847dc6f4cf16d935808"
Oct 30 22:45:40.358575 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.358526    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2329eccb1d9a8a74e2078baa52a15fc46a64f3648c5f3847dc6f4cf16d935808"} err="failed to get container status \"2329eccb1d9a8a74e2078baa52a15fc46a64f3648c5f3847dc6f4cf16d935808\": rpc error: code = NotFound desc = could not find container \"2329eccb1d9a8a74e2078baa52a15fc46a64f3648c5f3847dc6f4cf16d935808\": container with ID starting with 2329eccb1d9a8a74e2078baa52a15fc46a64f3648c5f3847dc6f4cf16d935808 not found: ID does not exist"
Oct 30 22:45:40.454061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.454015    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt67j\" (UniqueName: \"kubernetes.io/projected/0f3f457a-83fb-4bcd-a884-d9a988379046-kube-api-access-xt67j\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:40.454061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.454053    4553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3f457a-83fb-4bcd-a884-d9a988379046-tls-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:40.454061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.454068    4553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-dshm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:40.454435 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.454082    4553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-tmp-dir\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:40.454435 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.454095    4553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3f457a-83fb-4bcd-a884-d9a988379046-home\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:40.470261 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.470205    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" podStartSLOduration=2.4701903720000002 podStartE2EDuration="2.470190372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:40.467680471 +0000 UTC m=+3791.202043525" watchObservedRunningTime="2026-10-30 22:45:40.470190372 +0000 UTC m=+3791.204553403"
Oct 30 22:45:40.770351 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.770311    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"]
Oct 30 22:45:40.779187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:40.779129    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-89cf28/auth-test-model-89cf28-kserve-6dff76578f-6h7r9"]
Oct 30 22:45:41.250221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:41.250177    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:41.257327 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:41.257300    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:41.355074 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:41.355042    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:41.356280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:41.356254    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:45:41.394898 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:41.394862    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3f457a-83fb-4bcd-a884-d9a988379046" path="/var/lib/kubelet/pods/0f3f457a-83fb-4bcd-a884-d9a988379046/volumes"
Oct 30 22:45:52.128871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.128833    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"]
Oct 30 22:45:52.129587 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.129572    4553 cpu_manager.go:414] "containerMap: removing container" podUID="0f3f457a-83fb-4bcd-a884-d9a988379046" containerName="main"
Oct 30 22:45:52.129635 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.129589    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3f457a-83fb-4bcd-a884-d9a988379046" containerName="main"
Oct 30 22:45:52.129727 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.129718    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ddaa5a7-2fea-48d3-8698-1d4d4b222a65" containerName="cleanup"
Oct 30 22:45:52.129761 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.129736    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3f457a-83fb-4bcd-a884-d9a988379046" containerName="main"
Oct 30 22:45:52.134565 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.134548    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:52.201867 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.201826    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbrps\" (UniqueName: \"kubernetes.io/projected/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-kube-api-access-kbrps\") pod \"maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:52.202187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.202146    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:52.202301 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.202244    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-metrics-certs\") pod \"maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:52.202301 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.202295    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:52.239051 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.239018    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-55edb1-serving-cert\""
Oct 30 22:45:52.244726 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.244692    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"]
Oct 30 22:45:52.304108 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.304065    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:52.304308 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.304168    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-metrics-certs\") pod \"maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:52.304308 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.304206    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:52.304308 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.304271    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbrps\" (UniqueName: \"kubernetes.io/projected/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-kube-api-access-kbrps\") pod \"maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:52.304512 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:45:52.304379    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-serving-cert: secret "maas-api-e2e-extmod-a-55edb1-serving-cert" not found
Oct 30 22:45:52.304512 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:45:52.304468    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-maas-api-tls podName:7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5 nodeName:}" failed. No retries permitted until 2026-10-30 22:45:52.80444873 +0000 UTC m=+3803.538811758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-maas-api-tls") pod "maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh" (UID: "7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5") : secret "maas-api-e2e-extmod-a-55edb1-serving-cert" not found
Oct 30 22:45:52.304914 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.304887    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:52.307025 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.307001    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-metrics-certs\") pod \"maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:52.364927 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.364891    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbrps\" (UniqueName: \"kubernetes.io/projected/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-kube-api-access-kbrps\") pod \"maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:52.810632 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.810602    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:52.813375 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:52.813348    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:53.048796 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:53.048758    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:53.430917 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:53.430888    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"]
Oct 30 22:45:53.432124 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:45:53.432098    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7453d4cd_7f1a_42e0_a950_0fb3ca01aaf5.slice/crio-82688b0d4c097d470232f2c50a0eb571c47589c8e48447048a8a705f9d962ca0 WatchSource:0}: Error finding container 82688b0d4c097d470232f2c50a0eb571c47589c8e48447048a8a705f9d962ca0: Status 404 returned error can't find the container with id 82688b0d4c097d470232f2c50a0eb571c47589c8e48447048a8a705f9d962ca0
Oct 30 22:45:54.445691 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:54.445652    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh" event={"ID":"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5","Type":"ContainerStarted","Data":"82688b0d4c097d470232f2c50a0eb571c47589c8e48447048a8a705f9d962ca0"}
Oct 30 22:45:54.764600 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:54.764511    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9" podUID="d328e0bf-ad1f-4d08-a16d-f3e012187a85" containerName="main" containerID="cri-o://975fe2f536fed5569eb79a621f462b32c0ab52728864a957c1fe8a16b602d46d" gracePeriod=45
Oct 30 22:45:55.042760 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.042726    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:45:55.138680 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.138643    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-model-cache\") pod \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") "
Oct 30 22:45:55.138887 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.138842    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d328e0bf-ad1f-4d08-a16d-f3e012187a85-kube-api-access-2pzv8\" (UniqueName: \"kubernetes.io/projected/d328e0bf-ad1f-4d08-a16d-f3e012187a85-kube-api-access-2pzv8\") pod \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") "
Oct 30 22:45:55.138972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.138916    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-home\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-home\") pod \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") "
Oct 30 22:45:55.139033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.138966    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-dshm\") pod \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") "
Oct 30 22:45:55.139033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.138980    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-model-cache" pod "d328e0bf-ad1f-4d08-a16d-f3e012187a85" (UID: "d328e0bf-ad1f-4d08-a16d-f3e012187a85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:55.139033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.138994    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d328e0bf-ad1f-4d08-a16d-f3e012187a85-tls-certs\" (UniqueName: \"kubernetes.io/secret/d328e0bf-ad1f-4d08-a16d-f3e012187a85-tls-certs\") pod \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") "
Oct 30 22:45:55.139222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.139113    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-tmp-dir\") pod \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\" (UID: \"d328e0bf-ad1f-4d08-a16d-f3e012187a85\") "
Oct 30 22:45:55.139781 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.139618    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-home" pod "d328e0bf-ad1f-4d08-a16d-f3e012187a85" (UID: "d328e0bf-ad1f-4d08-a16d-f3e012187a85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:55.139781 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.139735    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-tmp-dir" pod "d328e0bf-ad1f-4d08-a16d-f3e012187a85" (UID: "d328e0bf-ad1f-4d08-a16d-f3e012187a85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:55.139988 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.139914    4553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-model-cache\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:55.139988 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.139935    4553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-home\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:55.141894 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.141849    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-dshm" pod "d328e0bf-ad1f-4d08-a16d-f3e012187a85" (UID: "d328e0bf-ad1f-4d08-a16d-f3e012187a85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:45:55.142029 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.142010    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d328e0bf-ad1f-4d08-a16d-f3e012187a85-kube-api-access-2pzv8" pod "d328e0bf-ad1f-4d08-a16d-f3e012187a85" (UID: "d328e0bf-ad1f-4d08-a16d-f3e012187a85"). InnerVolumeSpecName "kube-api-access-2pzv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:55.142307 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.142283    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d328e0bf-ad1f-4d08-a16d-f3e012187a85-tls-certs" pod "d328e0bf-ad1f-4d08-a16d-f3e012187a85" (UID: "d328e0bf-ad1f-4d08-a16d-f3e012187a85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:55.240957 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.240918    4553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-tmp-dir\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:55.240957 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.240952    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pzv8\" (UniqueName: \"kubernetes.io/projected/d328e0bf-ad1f-4d08-a16d-f3e012187a85-kube-api-access-2pzv8\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:55.240957 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.240963    4553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d328e0bf-ad1f-4d08-a16d-f3e012187a85-dshm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:55.241250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.240973    4553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d328e0bf-ad1f-4d08-a16d-f3e012187a85-tls-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:55.453601 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.453564    4553 generic.go:356] "Generic (PLEG): container finished" podID="d328e0bf-ad1f-4d08-a16d-f3e012187a85" containerID="975fe2f536fed5569eb79a621f462b32c0ab52728864a957c1fe8a16b602d46d" exitCode=0
Oct 30 22:45:55.454067 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.453618    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"
Oct 30 22:45:55.454067 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.453651    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9" event={"ID":"d328e0bf-ad1f-4d08-a16d-f3e012187a85","Type":"ContainerDied","Data":"975fe2f536fed5569eb79a621f462b32c0ab52728864a957c1fe8a16b602d46d"}
Oct 30 22:45:55.454067 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.453693    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9" event={"ID":"d328e0bf-ad1f-4d08-a16d-f3e012187a85","Type":"ContainerDied","Data":"9f8f5b61681f648f2246d62105a3b279c270b1b4d278729d8461a9195dbc4307"}
Oct 30 22:45:55.454067 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.453709    4553 scope.go:122] "RemoveContainer" containerID="975fe2f536fed5569eb79a621f462b32c0ab52728864a957c1fe8a16b602d46d"
Oct 30 22:45:55.455764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.455734    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh" event={"ID":"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5","Type":"ContainerStarted","Data":"74938b926a147dce4e1df3b5b206539881272cecb9d3ed646bd194396d99125a"}
Oct 30 22:45:55.455949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.455905    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:45:55.464456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.464435    4553 scope.go:122] "RemoveContainer" containerID="975fe2f536fed5569eb79a621f462b32c0ab52728864a957c1fe8a16b602d46d"
Oct 30 22:45:55.464743 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:45:55.464724    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975fe2f536fed5569eb79a621f462b32c0ab52728864a957c1fe8a16b602d46d\": container with ID starting with 975fe2f536fed5569eb79a621f462b32c0ab52728864a957c1fe8a16b602d46d not found: ID does not exist" containerID="975fe2f536fed5569eb79a621f462b32c0ab52728864a957c1fe8a16b602d46d"
Oct 30 22:45:55.464810 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.464757    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975fe2f536fed5569eb79a621f462b32c0ab52728864a957c1fe8a16b602d46d"} err="failed to get container status \"975fe2f536fed5569eb79a621f462b32c0ab52728864a957c1fe8a16b602d46d\": rpc error: code = NotFound desc = could not find container \"975fe2f536fed5569eb79a621f462b32c0ab52728864a957c1fe8a16b602d46d\": container with ID starting with 975fe2f536fed5569eb79a621f462b32c0ab52728864a957c1fe8a16b602d46d not found: ID does not exist"
Oct 30 22:45:55.623837 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.623789    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh" podStartSLOduration=2.686836381 podStartE2EDuration="3.623774995s" totalImagesPullingTime="936.938614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:55.623225151 +0000 UTC m=+3806.357588181" watchObservedRunningTime="2026-10-30 22:45:55.623774995 +0000 UTC m=+3806.358138023"
Oct 30 22:45:55.740069 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.739986    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"]
Oct 30 22:45:55.743714 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:55.743685    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0e450a/auth-test-model-0e450a-kserve-6f5bf5d565-lftc9"]
Oct 30 22:45:57.398933 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:45:57.398894    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d328e0bf-ad1f-4d08-a16d-f3e012187a85" path="/var/lib/kubelet/pods/d328e0bf-ad1f-4d08-a16d-f3e012187a85/volumes"
Oct 30 22:45:58.202254 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:45:58.202219    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:46:01.469315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:01.469287    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:46:09.685953 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:09.685868    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"]
Oct 30 22:46:09.686689 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:09.686671    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d328e0bf-ad1f-4d08-a16d-f3e012187a85" containerName="main"
Oct 30 22:46:09.686753 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:09.686724    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d328e0bf-ad1f-4d08-a16d-f3e012187a85" containerName="main"
Oct 30 22:46:09.686874 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:09.686863    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d328e0bf-ad1f-4d08-a16d-f3e012187a85" containerName="main"
Oct 30 22:46:09.690621 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:09.690598    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"
Oct 30 22:46:09.729257 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:09.729222    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whzr2\" (UniqueName: \"kubernetes.io/projected/0d13c173-2f05-47f6-837d-4e09f3984114-kube-api-access-whzr2\") pod \"payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg\" (UID: \"0d13c173-2f05-47f6-837d-4e09f3984114\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"
Oct 30 22:46:09.729461 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:09.729310    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d13c173-2f05-47f6-837d-4e09f3984114-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg\" (UID: \"0d13c173-2f05-47f6-837d-4e09f3984114\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"
Oct 30 22:46:09.746259 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:09.746227    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"]
Oct 30 22:46:09.792980 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:09.792942    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-9e18b5\""
Oct 30 22:46:09.792980 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:09.792951    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-9e18b5-dockercfg-snknj\""
Oct 30 22:46:09.832003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:09.831967    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whzr2\" (UniqueName: \"kubernetes.io/projected/0d13c173-2f05-47f6-837d-4e09f3984114-kube-api-access-whzr2\") pod \"payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg\" (UID: \"0d13c173-2f05-47f6-837d-4e09f3984114\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"
Oct 30 22:46:09.833583 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:09.833554    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d13c173-2f05-47f6-837d-4e09f3984114-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg\" (UID: \"0d13c173-2f05-47f6-837d-4e09f3984114\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"
Oct 30 22:46:09.836428 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:09.836390    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d13c173-2f05-47f6-837d-4e09f3984114-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg\" (UID: \"0d13c173-2f05-47f6-837d-4e09f3984114\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"
Oct 30 22:46:09.891341 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:09.891309    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whzr2\" (UniqueName: \"kubernetes.io/projected/0d13c173-2f05-47f6-837d-4e09f3984114-kube-api-access-whzr2\") pod \"payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg\" (UID: \"0d13c173-2f05-47f6-837d-4e09f3984114\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"
Oct 30 22:46:10.004787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:10.004681    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"
Oct 30 22:46:10.387658 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:10.387628    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"]
Oct 30 22:46:10.388353 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:46:10.388323    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d13c173_2f05_47f6_837d_4e09f3984114.slice/crio-336dda69df482a1e6be3cf5fc36e323c79309a3d0452dc133792e0b82719f983 WatchSource:0}: Error finding container 336dda69df482a1e6be3cf5fc36e323c79309a3d0452dc133792e0b82719f983: Status 404 returned error can't find the container with id 336dda69df482a1e6be3cf5fc36e323c79309a3d0452dc133792e0b82719f983
Oct 30 22:46:10.562013 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:10.561975    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg" event={"ID":"0d13c173-2f05-47f6-837d-4e09f3984114","Type":"ContainerStarted","Data":"2c2f7ca6034ac80b7ca18f1cc5bbc39de3f44ce1d9fdd379537c3b5677f854f1"}
Oct 30 22:46:10.562246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:10.562039    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg" event={"ID":"0d13c173-2f05-47f6-837d-4e09f3984114","Type":"ContainerStarted","Data":"336dda69df482a1e6be3cf5fc36e323c79309a3d0452dc133792e0b82719f983"}
Oct 30 22:46:10.562246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:10.562195    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"
Oct 30 22:46:10.679770 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:10.679653    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg" podStartSLOduration=1.6796363429999999 podStartE2EDuration="1.679636343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:10.678662876 +0000 UTC m=+3821.413025908" watchObservedRunningTime="2026-10-30 22:46:10.679636343 +0000 UTC m=+3821.413999374"
Oct 30 22:46:21.572187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:21.572157    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"
Oct 30 22:46:26.180442 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.180401    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"]
Oct 30 22:46:26.185926 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.185901    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.243111 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.243032    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b1b1eb-openshift-default-dockercfg-hvjj7\""
Oct 30 22:46:26.299157 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.298916    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"]
Oct 30 22:46:26.340372 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.340332    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4a2de90-2ce9-452e-a5ed-63606a25e513-istiod-ca-cert\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.340655 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.340387    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-workload-socket\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.340655 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.340468    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-envoy\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.340655 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.340506    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-podinfo\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.340655 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.340535    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-workload-certs\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.340655 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.340601    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-data\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.340655 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.340657    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-token\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.341188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.340706    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj4bg\" (UniqueName: \"kubernetes.io/projected/b4a2de90-2ce9-452e-a5ed-63606a25e513-kube-api-access-kj4bg\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.341188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.340884    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-credential-socket\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.441746 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.441662    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-credential-socket\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.441746 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.441704    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4a2de90-2ce9-452e-a5ed-63606a25e513-istiod-ca-cert\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.441746 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.441725    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-workload-socket\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.442061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.441779    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-envoy\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.442061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.441800    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-podinfo\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.442061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.441825    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-workload-certs\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.442061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.441881    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-data\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.442061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.442040    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-token\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.442368 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.442105    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4bg\" (UniqueName: \"kubernetes.io/projected/b4a2de90-2ce9-452e-a5ed-63606a25e513-kube-api-access-kj4bg\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.442368 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.442279    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-data\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.442368 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.442310    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-workload-socket\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.442368 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.442361    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-workload-certs\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.442731 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.442616    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-credential-socket\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.442731 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.442733    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4a2de90-2ce9-452e-a5ed-63606a25e513-istiod-ca-cert\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.444456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.444427    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-envoy\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.444567 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.444537    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-podinfo\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.503642 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.503606    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-token\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.503797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.503682    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj4bg\" (UniqueName: \"kubernetes.io/projected/b4a2de90-2ce9-452e-a5ed-63606a25e513-kube-api-access-kj4bg\") pod \"e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") " pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:26.801265 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:26.801168    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:27.237107 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:27.237079    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"]
Oct 30 22:46:27.238286 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:46:27.238259    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a2de90_2ce9_452e_a5ed_63606a25e513.slice/crio-02a1205f02a60959f65025e01534e68bb6fb1ed6f1a4c9eec781dc17b3e967e1 WatchSource:0}: Error finding container 02a1205f02a60959f65025e01534e68bb6fb1ed6f1a4c9eec781dc17b3e967e1: Status 404 returned error can't find the container with id 02a1205f02a60959f65025e01534e68bb6fb1ed6f1a4c9eec781dc17b3e967e1
Oct 30 22:46:27.241181 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:27.241151    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:46:27.241267 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:27.241225    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:46:27.241267 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:27.241255    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:46:27.683355 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:27.683322    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4" event={"ID":"b4a2de90-2ce9-452e-a5ed-63606a25e513","Type":"ContainerStarted","Data":"038c8522d1a99a4d19c4c82931bfca77d880ae97b3668c195fa5571d6b0b04a6"}
Oct 30 22:46:27.683557 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:27.683377    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4" event={"ID":"b4a2de90-2ce9-452e-a5ed-63606a25e513","Type":"ContainerStarted","Data":"02a1205f02a60959f65025e01534e68bb6fb1ed6f1a4c9eec781dc17b3e967e1"}
Oct 30 22:46:27.802016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:27.801984    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:27.807351 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:27.807298    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4" podStartSLOduration=1.807284734 podStartE2EDuration="1.807284734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:27.806502224 +0000 UTC m=+3838.540865269" watchObservedRunningTime="2026-10-30 22:46:27.807284734 +0000 UTC m=+3838.541647765"
Oct 30 22:46:28.204119 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:46:28.204080    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:46:28.808570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:28.808537    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:29.697432 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:29.697400    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:29.698696 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:46:29.698677    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:46:58.203048 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:46:58.202999    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:47:17.502313 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:17.502281    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"]
Oct 30 22:47:17.514724 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:17.514684    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:17.564615 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:17.564583    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"]
Oct 30 22:47:17.610722 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:17.610688    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-32af64-serving-cert\""
Oct 30 22:47:17.618615 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:17.618586    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-metrics-certs\") pod \"maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:17.618750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:17.618625    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:17.618750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:17.618664    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:17.618824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:17.618790    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s2gz\" (UniqueName: \"kubernetes.io/projected/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-kube-api-access-7s2gz\") pod \"maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:17.720406 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:17.720366    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-metrics-certs\") pod \"maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:17.720406 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:17.720407    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:17.720714 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:47:17.720527    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-serving-cert: secret "maas-api-e2e-extmod-b-32af64-serving-cert" not found
Oct 30 22:47:17.720714 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:17.720540    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:17.720714 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:47:17.720606    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-maas-api-tls podName:226981d5-7cad-4f8b-9a87-e3c6bef2e59a nodeName:}" failed. No retries permitted until 2026-10-30 22:47:18.220585129 +0000 UTC m=+3888.954948155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-maas-api-tls") pod "maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm" (UID: "226981d5-7cad-4f8b-9a87-e3c6bef2e59a") : secret "maas-api-e2e-extmod-b-32af64-serving-cert" not found
Oct 30 22:47:17.720714 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:17.720643    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s2gz\" (UniqueName: \"kubernetes.io/projected/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-kube-api-access-7s2gz\") pod \"maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:17.721296 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:17.721275    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:17.723319 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:17.723285    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-metrics-certs\") pod \"maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:17.779739 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:17.779668    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s2gz\" (UniqueName: \"kubernetes.io/projected/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-kube-api-access-7s2gz\") pod \"maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:18.227212 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:18.227173    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:18.229805 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:18.229781    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-maas-api-tls\") pod \"maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:18.429640 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:18.429604    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:18.841651 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:18.841618    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"]
Oct 30 22:47:18.842508 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:47:18.842479    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226981d5_7cad_4f8b_9a87_e3c6bef2e59a.slice/crio-622de89e36c1dd24b2e7900f65fe60294019ff77c2adc246ddc78c3fe1fbfdee WatchSource:0}: Error finding container 622de89e36c1dd24b2e7900f65fe60294019ff77c2adc246ddc78c3fe1fbfdee: Status 404 returned error can't find the container with id 622de89e36c1dd24b2e7900f65fe60294019ff77c2adc246ddc78c3fe1fbfdee
Oct 30 22:47:19.037118 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:19.037083    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm" event={"ID":"226981d5-7cad-4f8b-9a87-e3c6bef2e59a","Type":"ContainerStarted","Data":"622de89e36c1dd24b2e7900f65fe60294019ff77c2adc246ddc78c3fe1fbfdee"}
Oct 30 22:47:20.045113 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:20.045078    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm" event={"ID":"226981d5-7cad-4f8b-9a87-e3c6bef2e59a","Type":"ContainerStarted","Data":"b14296b50f51eaef29063890dd88dc505ecac7ee335c4af85f6ae83bdd181cf2"}
Oct 30 22:47:20.045576 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:20.045156    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:26.059216 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:26.059185    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:47:26.179869 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:26.179817    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm" podStartSLOduration=8.223441524 podStartE2EDuration="9.179802416s" totalImagesPullingTime="956.360892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:20.161820741 +0000 UTC m=+3890.896183772" watchObservedRunningTime="2026-10-30 22:47:26.179802416 +0000 UTC m=+3896.914165447"
Oct 30 22:47:28.203196 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:47:28.203114    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:47:30.052606 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:30.052581    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:47:30.061809 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:30.061782    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:47:34.558342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:34.558299    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"]
Oct 30 22:47:39.559073 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:39.559000    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f" podUID="51090a1f-1c13-4daa-b493-1cb54373cb43" containerName="payload-processing" containerID="cri-o://52e361dfba7742579f30a1d7967176043800d6b9095de6e35bd9423409ec46c4" gracePeriod=25
Oct 30 22:47:39.881350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:39.881326    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"
Oct 30 22:47:40.043824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:40.043781    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51090a1f-1c13-4daa-b493-1cb54373cb43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51090a1f-1c13-4daa-b493-1cb54373cb43-plugins-config-volume\") pod \"51090a1f-1c13-4daa-b493-1cb54373cb43\" (UID: \"51090a1f-1c13-4daa-b493-1cb54373cb43\") "
Oct 30 22:47:40.044146 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:40.044107    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51090a1f-1c13-4daa-b493-1cb54373cb43-kube-api-access-vlgnc\" (UniqueName: \"kubernetes.io/projected/51090a1f-1c13-4daa-b493-1cb54373cb43-kube-api-access-vlgnc\") pod \"51090a1f-1c13-4daa-b493-1cb54373cb43\" (UID: \"51090a1f-1c13-4daa-b493-1cb54373cb43\") "
Oct 30 22:47:40.044270 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:40.044186    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51090a1f-1c13-4daa-b493-1cb54373cb43-plugins-config-volume" pod "51090a1f-1c13-4daa-b493-1cb54373cb43" (UID: "51090a1f-1c13-4daa-b493-1cb54373cb43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:40.044690 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:40.044668    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51090a1f-1c13-4daa-b493-1cb54373cb43-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:40.046952 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:40.046917    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51090a1f-1c13-4daa-b493-1cb54373cb43-kube-api-access-vlgnc" pod "51090a1f-1c13-4daa-b493-1cb54373cb43" (UID: "51090a1f-1c13-4daa-b493-1cb54373cb43"). InnerVolumeSpecName "kube-api-access-vlgnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:40.145782 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:40.145694    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlgnc\" (UniqueName: \"kubernetes.io/projected/51090a1f-1c13-4daa-b493-1cb54373cb43-kube-api-access-vlgnc\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:47:40.184302 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:40.184264    4553 generic.go:356] "Generic (PLEG): container finished" podID="51090a1f-1c13-4daa-b493-1cb54373cb43" containerID="52e361dfba7742579f30a1d7967176043800d6b9095de6e35bd9423409ec46c4" exitCode=0
Oct 30 22:47:40.184485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:40.184310    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"
Oct 30 22:47:40.184485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:40.184326    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f" event={"ID":"51090a1f-1c13-4daa-b493-1cb54373cb43","Type":"ContainerDied","Data":"52e361dfba7742579f30a1d7967176043800d6b9095de6e35bd9423409ec46c4"}
Oct 30 22:47:40.184485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:40.184385    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f" event={"ID":"51090a1f-1c13-4daa-b493-1cb54373cb43","Type":"ContainerDied","Data":"ee54ff90832d1478328dc5b37bbb4e93a9d5f67759ab6acdf06fc91ba71e72c7"}
Oct 30 22:47:40.184485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:40.184413    4553 scope.go:122] "RemoveContainer" containerID="52e361dfba7742579f30a1d7967176043800d6b9095de6e35bd9423409ec46c4"
Oct 30 22:47:40.195799 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:40.195780    4553 scope.go:122] "RemoveContainer" containerID="52e361dfba7742579f30a1d7967176043800d6b9095de6e35bd9423409ec46c4"
Oct 30 22:47:40.196045 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:47:40.196024    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52e361dfba7742579f30a1d7967176043800d6b9095de6e35bd9423409ec46c4\": container with ID starting with 52e361dfba7742579f30a1d7967176043800d6b9095de6e35bd9423409ec46c4 not found: ID does not exist" containerID="52e361dfba7742579f30a1d7967176043800d6b9095de6e35bd9423409ec46c4"
Oct 30 22:47:40.196121 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:40.196051    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52e361dfba7742579f30a1d7967176043800d6b9095de6e35bd9423409ec46c4"} err="failed to get container status \"52e361dfba7742579f30a1d7967176043800d6b9095de6e35bd9423409ec46c4\": rpc error: code = NotFound desc = could not find container \"52e361dfba7742579f30a1d7967176043800d6b9095de6e35bd9423409ec46c4\": container with ID starting with 52e361dfba7742579f30a1d7967176043800d6b9095de6e35bd9423409ec46c4 not found: ID does not exist"
Oct 30 22:47:40.352055 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:40.352012    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"]
Oct 30 22:47:40.356340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:40.356316    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-55edb1-86677cc887-fmc2f"]
Oct 30 22:47:41.396451 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:41.396405    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51090a1f-1c13-4daa-b493-1cb54373cb43" path="/var/lib/kubelet/pods/51090a1f-1c13-4daa-b493-1cb54373cb43/volumes"
Oct 30 22:47:42.397363 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:42.397313    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:47:42.397789 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:42.397374    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:47:45.224546 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:45.224514    4553 generic.go:356] "Generic (PLEG): container finished" podID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerID="3e931b688cc5dd3cde0f474d1373e89336409217d59519d2bdd4ecdcf49ffd1d" exitCode=0
Oct 30 22:47:45.225064 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:45.224598    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" event={"ID":"d5f83e3e-757f-495d-9734-e3f7a77d9b9d","Type":"ContainerDied","Data":"3e931b688cc5dd3cde0f474d1373e89336409217d59519d2bdd4ecdcf49ffd1d"}
Oct 30 22:47:45.225064 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:45.225022    4553 scope.go:122] "RemoveContainer" containerID="3e931b688cc5dd3cde0f474d1373e89336409217d59519d2bdd4ecdcf49ffd1d"
Oct 30 22:47:45.226764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:45.226736    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:47:45.226842 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:45.226808    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:47:45.226882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:45.226841    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:47:46.242096 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:46.242056    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" event={"ID":"d5f83e3e-757f-495d-9734-e3f7a77d9b9d","Type":"ContainerStarted","Data":"3fbda9fb903695a0a59557e5484a63289534070470c8870001aba1d20aaafa3d"}
Oct 30 22:47:46.250679 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:46.250650    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:47:46.250859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:46.250828    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 30 22:47:46.250921 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:46.250882    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 30 22:47:47.251034 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.250994    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 30 22:47:47.251570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.251051    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 30 22:47:47.306370 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.306338    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"]
Oct 30 22:47:47.307164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.307117    4553 cpu_manager.go:414] "containerMap: removing container" podUID="51090a1f-1c13-4daa-b493-1cb54373cb43" containerName="payload-processing"
Oct 30 22:47:47.307164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.307164    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51090a1f-1c13-4daa-b493-1cb54373cb43" containerName="payload-processing"
Oct 30 22:47:47.307425 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.307409    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="51090a1f-1c13-4daa-b493-1cb54373cb43" containerName="payload-processing"
Oct 30 22:47:47.334580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.334550    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:47.349943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.349910    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-metrics-certs\") pod \"maas-api-e2e-autores-9e18b5-5c88788b5-4bccc\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:47.350101 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.350031    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvr5h\" (UniqueName: \"kubernetes.io/projected/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-kube-api-access-hvr5h\") pod \"maas-api-e2e-autores-9e18b5-5c88788b5-4bccc\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:47.350200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.350101    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-maas-api-tls\") pod \"maas-api-e2e-autores-9e18b5-5c88788b5-4bccc\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:47.350385 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.350363    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-openshift-service-ca\") pod \"maas-api-e2e-autores-9e18b5-5c88788b5-4bccc\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:47.368825 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.368796    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"]
Oct 30 22:47:47.433164 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.432289    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-9e18b5-serving-cert\""
Oct 30 22:47:47.451580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.451549    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-metrics-certs\") pod \"maas-api-e2e-autores-9e18b5-5c88788b5-4bccc\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:47.451763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.451651    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvr5h\" (UniqueName: \"kubernetes.io/projected/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-kube-api-access-hvr5h\") pod \"maas-api-e2e-autores-9e18b5-5c88788b5-4bccc\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:47.451763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.451709    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-maas-api-tls\") pod \"maas-api-e2e-autores-9e18b5-5c88788b5-4bccc\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:47.451763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.451749    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-openshift-service-ca\") pod \"maas-api-e2e-autores-9e18b5-5c88788b5-4bccc\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:47.453409 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:47:47.452525    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-serving-cert: secret "maas-api-e2e-autores-9e18b5-serving-cert" not found
Oct 30 22:47:47.453409 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:47:47.452590    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-maas-api-tls podName:2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf nodeName:}" failed. No retries permitted until 2026-10-30 22:47:47.952571625 +0000 UTC m=+3918.686934648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-maas-api-tls") pod "maas-api-e2e-autores-9e18b5-5c88788b5-4bccc" (UID: "2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf") : secret "maas-api-e2e-autores-9e18b5-serving-cert" not found
Oct 30 22:47:47.453409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.452626    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-openshift-service-ca\") pod \"maas-api-e2e-autores-9e18b5-5c88788b5-4bccc\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:47.455574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.455550    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-metrics-certs\") pod \"maas-api-e2e-autores-9e18b5-5c88788b5-4bccc\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:47.534094 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.534010    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvr5h\" (UniqueName: \"kubernetes.io/projected/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-kube-api-access-hvr5h\") pod \"maas-api-e2e-autores-9e18b5-5c88788b5-4bccc\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:47.958482 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.958448    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-maas-api-tls\") pod \"maas-api-e2e-autores-9e18b5-5c88788b5-4bccc\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:47.961182 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:47.961156    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-maas-api-tls\") pod \"maas-api-e2e-autores-9e18b5-5c88788b5-4bccc\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:48.250287 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:48.250145    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:48.250456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:48.250302    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 30 22:47:48.250456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:48.250395    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 30 22:47:48.647964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:48.647933    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"]
Oct 30 22:47:48.648955 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:47:48.648931    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2715ecd9_7cfe_448f_9cd6_0899e0e8e5bf.slice/crio-dc4ab2f9dd1ee2ab61491b06f430f7f28558b2578e68c71138184a84bd6fea4a WatchSource:0}: Error finding container dc4ab2f9dd1ee2ab61491b06f430f7f28558b2578e68c71138184a84bd6fea4a: Status 404 returned error can't find the container with id dc4ab2f9dd1ee2ab61491b06f430f7f28558b2578e68c71138184a84bd6fea4a
Oct 30 22:47:49.250429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:49.250392    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 30 22:47:49.250614 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:49.250444    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 30 22:47:49.263069 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:49.263035    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc" event={"ID":"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf","Type":"ContainerStarted","Data":"dc4ab2f9dd1ee2ab61491b06f430f7f28558b2578e68c71138184a84bd6fea4a"}
Oct 30 22:47:50.250730 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:50.250689    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 30 22:47:50.251203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:50.250746    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 30 22:47:50.271584 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:50.271550    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc" event={"ID":"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf","Type":"ContainerStarted","Data":"efb5439848362f662c73a60c9a19a091008d77909a075e913e933b501eaf26c6"}
Oct 30 22:47:50.271750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:50.271651    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:50.435170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:50.431389    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc" podStartSLOduration=2.329971299 podStartE2EDuration="3.431369937s" totalImagesPullingTime="1.101398638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:50.430453687 +0000 UTC m=+3921.164816720" watchObservedRunningTime="2026-10-30 22:47:50.431369937 +0000 UTC m=+3921.165732970"
Oct 30 22:47:51.250286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:51.250251    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Oct 30 22:47:51.250477 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:51.250315    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Oct 30 22:47:52.254485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:52.254455    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:47:52.254955 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:52.254936    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:47:52.255708 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:52.255691    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:47:56.285190 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:47:56.285127    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:47:58.202195 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:47:58.202156    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:48:28.205576 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:48:28.205540    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:48:29.144112 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:29.144072    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"]
Oct 30 22:48:34.145158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.145096    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br" podUID="843153d0-0973-45a5-b4bc-063f6ea52968" containerName="payload-pre-processing" containerID="cri-o://7a6d92250237c240aaa27d1c9b9cd6556f83e024bb3189f5d2c1e1d6112bb5cf" gracePeriod=25
Oct 30 22:48:34.413797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.413731    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"
Oct 30 22:48:34.494652 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.494620    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/843153d0-0973-45a5-b4bc-063f6ea52968-kube-api-access-7j6f8\" (UniqueName: \"kubernetes.io/projected/843153d0-0973-45a5-b4bc-063f6ea52968-kube-api-access-7j6f8\") pod \"843153d0-0973-45a5-b4bc-063f6ea52968\" (UID: \"843153d0-0973-45a5-b4bc-063f6ea52968\") "
Oct 30 22:48:34.494832 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.494724    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/843153d0-0973-45a5-b4bc-063f6ea52968-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/843153d0-0973-45a5-b4bc-063f6ea52968-plugins-config-volume\") pod \"843153d0-0973-45a5-b4bc-063f6ea52968\" (UID: \"843153d0-0973-45a5-b4bc-063f6ea52968\") "
Oct 30 22:48:34.495202 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.495171    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/843153d0-0973-45a5-b4bc-063f6ea52968-plugins-config-volume" pod "843153d0-0973-45a5-b4bc-063f6ea52968" (UID: "843153d0-0973-45a5-b4bc-063f6ea52968"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:34.495569 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.495543    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/843153d0-0973-45a5-b4bc-063f6ea52968-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:34.497038 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.497019    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843153d0-0973-45a5-b4bc-063f6ea52968-kube-api-access-7j6f8" pod "843153d0-0973-45a5-b4bc-063f6ea52968" (UID: "843153d0-0973-45a5-b4bc-063f6ea52968"). InnerVolumeSpecName "kube-api-access-7j6f8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:34.581420 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.581386    4553 generic.go:356] "Generic (PLEG): container finished" podID="843153d0-0973-45a5-b4bc-063f6ea52968" containerID="7a6d92250237c240aaa27d1c9b9cd6556f83e024bb3189f5d2c1e1d6112bb5cf" exitCode=0
Oct 30 22:48:34.581420 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.581429    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"
Oct 30 22:48:34.581659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.581469    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br" event={"ID":"843153d0-0973-45a5-b4bc-063f6ea52968","Type":"ContainerDied","Data":"7a6d92250237c240aaa27d1c9b9cd6556f83e024bb3189f5d2c1e1d6112bb5cf"}
Oct 30 22:48:34.581659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.581527    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br" event={"ID":"843153d0-0973-45a5-b4bc-063f6ea52968","Type":"ContainerDied","Data":"207a6cd5e9ae096235862073e6a77d2f255d479615a49455bf207a009498ea55"}
Oct 30 22:48:34.581659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.581550    4553 scope.go:122] "RemoveContainer" containerID="7a6d92250237c240aaa27d1c9b9cd6556f83e024bb3189f5d2c1e1d6112bb5cf"
Oct 30 22:48:34.592972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.592953    4553 scope.go:122] "RemoveContainer" containerID="7a6d92250237c240aaa27d1c9b9cd6556f83e024bb3189f5d2c1e1d6112bb5cf"
Oct 30 22:48:34.593421 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:48:34.593397    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6d92250237c240aaa27d1c9b9cd6556f83e024bb3189f5d2c1e1d6112bb5cf\": container with ID starting with 7a6d92250237c240aaa27d1c9b9cd6556f83e024bb3189f5d2c1e1d6112bb5cf not found: ID does not exist" containerID="7a6d92250237c240aaa27d1c9b9cd6556f83e024bb3189f5d2c1e1d6112bb5cf"
Oct 30 22:48:34.593490 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.593431    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6d92250237c240aaa27d1c9b9cd6556f83e024bb3189f5d2c1e1d6112bb5cf"} err="failed to get container status \"7a6d92250237c240aaa27d1c9b9cd6556f83e024bb3189f5d2c1e1d6112bb5cf\": rpc error: code = NotFound desc = could not find container \"7a6d92250237c240aaa27d1c9b9cd6556f83e024bb3189f5d2c1e1d6112bb5cf\": container with ID starting with 7a6d92250237c240aaa27d1c9b9cd6556f83e024bb3189f5d2c1e1d6112bb5cf not found: ID does not exist"
Oct 30 22:48:34.597245 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.597225    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j6f8\" (UniqueName: \"kubernetes.io/projected/843153d0-0973-45a5-b4bc-063f6ea52968-kube-api-access-7j6f8\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:48:34.748349 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.748258    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"]
Oct 30 22:48:34.751430 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:34.751407    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a6900c-688d86d78c-c26br"]
Oct 30 22:48:35.394090 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:35.394057    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="843153d0-0973-45a5-b4bc-063f6ea52968" path="/var/lib/kubelet/pods/843153d0-0973-45a5-b4bc-063f6ea52968/volumes"
Oct 30 22:48:54.724841 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:54.724804    4553 generic.go:356] "Generic (PLEG): container finished" podID="b4a2de90-2ce9-452e-a5ed-63606a25e513" containerID="038c8522d1a99a4d19c4c82931bfca77d880ae97b3668c195fa5571d6b0b04a6" exitCode=0
Oct 30 22:48:54.725434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:54.724882    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4" event={"ID":"b4a2de90-2ce9-452e-a5ed-63606a25e513","Type":"ContainerDied","Data":"038c8522d1a99a4d19c4c82931bfca77d880ae97b3668c195fa5571d6b0b04a6"}
Oct 30 22:48:54.725509 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:54.725454    4553 scope.go:122] "RemoveContainer" containerID="038c8522d1a99a4d19c4c82931bfca77d880ae97b3668c195fa5571d6b0b04a6"
Oct 30 22:48:54.727287 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:54.727258    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:48:54.727405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:54.727346    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:48:54.727405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:54.727393    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:48:55.746727 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:55.746685    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4" event={"ID":"b4a2de90-2ce9-452e-a5ed-63606a25e513","Type":"ContainerStarted","Data":"6f50dd5e5c5b97a46765121d3b2679038ef8ee212f77288a2eace1bf1de4a904"}
Oct 30 22:48:55.801698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:55.801657    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:48:55.803251 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:55.803223    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:48:55.803384 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:55.803276    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4" podUID="b4a2de90-2ce9-452e-a5ed-63606a25e513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 30 22:48:56.801808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:56.801771    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:48:56.802346 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:56.801824    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4" podUID="b4a2de90-2ce9-452e-a5ed-63606a25e513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 30 22:48:57.802123 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:57.802090    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:48:57.802574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:57.802174    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4" podUID="b4a2de90-2ce9-452e-a5ed-63606a25e513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 30 22:48:58.203378 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:48:58.203343    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:48:58.235087 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:58.235045    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"]
Oct 30 22:48:58.802153 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:58.802112    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:48:58.802585 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:58.802204    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4" podUID="b4a2de90-2ce9-452e-a5ed-63606a25e513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 30 22:48:59.698171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:59.698127    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:48:59.802360 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:59.802330    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:48:59.802851 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:48:59.802386    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4" podUID="b4a2de90-2ce9-452e-a5ed-63606a25e513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 30 22:49:00.801938 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:00.801899    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Oct 30 22:49:00.802187 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:00.801954    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4" podUID="b4a2de90-2ce9-452e-a5ed-63606a25e513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Oct 30 22:49:01.806796 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:01.806760    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:49:01.807835 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:01.807809    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:49:03.235986 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.235940    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx" podUID="25abbe77-bfce-472d-84f4-2d5d9bff0c4d" containerName="payload-processing" containerID="cri-o://beadfa8b8089d06f0ac84a218802270351e6def1a4f37b76f77fc7ee9bce8823" gracePeriod=25
Oct 30 22:49:03.517627 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.517604    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"
Oct 30 22:49:03.632711 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.632670    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25abbe77-bfce-472d-84f4-2d5d9bff0c4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25abbe77-bfce-472d-84f4-2d5d9bff0c4d-plugins-config-volume\") pod \"25abbe77-bfce-472d-84f4-2d5d9bff0c4d\" (UID: \"25abbe77-bfce-472d-84f4-2d5d9bff0c4d\") "
Oct 30 22:49:03.633272 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.633179    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25abbe77-bfce-472d-84f4-2d5d9bff0c4d-kube-api-access-dlt82\" (UniqueName: \"kubernetes.io/projected/25abbe77-bfce-472d-84f4-2d5d9bff0c4d-kube-api-access-dlt82\") pod \"25abbe77-bfce-472d-84f4-2d5d9bff0c4d\" (UID: \"25abbe77-bfce-472d-84f4-2d5d9bff0c4d\") "
Oct 30 22:49:03.633272 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.633196    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25abbe77-bfce-472d-84f4-2d5d9bff0c4d-plugins-config-volume" pod "25abbe77-bfce-472d-84f4-2d5d9bff0c4d" (UID: "25abbe77-bfce-472d-84f4-2d5d9bff0c4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:03.633844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.633812    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25abbe77-bfce-472d-84f4-2d5d9bff0c4d-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:03.636392 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.636364    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25abbe77-bfce-472d-84f4-2d5d9bff0c4d-kube-api-access-dlt82" pod "25abbe77-bfce-472d-84f4-2d5d9bff0c4d" (UID: "25abbe77-bfce-472d-84f4-2d5d9bff0c4d"). InnerVolumeSpecName "kube-api-access-dlt82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:03.735303 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.735271    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlt82\" (UniqueName: \"kubernetes.io/projected/25abbe77-bfce-472d-84f4-2d5d9bff0c4d-kube-api-access-dlt82\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:03.809588 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.809499    4553 generic.go:356] "Generic (PLEG): container finished" podID="25abbe77-bfce-472d-84f4-2d5d9bff0c4d" containerID="beadfa8b8089d06f0ac84a218802270351e6def1a4f37b76f77fc7ee9bce8823" exitCode=0
Oct 30 22:49:03.809588 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.809549    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"
Oct 30 22:49:03.809588 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.809574    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx" event={"ID":"25abbe77-bfce-472d-84f4-2d5d9bff0c4d","Type":"ContainerDied","Data":"beadfa8b8089d06f0ac84a218802270351e6def1a4f37b76f77fc7ee9bce8823"}
Oct 30 22:49:03.809865 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.809626    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx" event={"ID":"25abbe77-bfce-472d-84f4-2d5d9bff0c4d","Type":"ContainerDied","Data":"e23e88d4bd9e53bf5f8299337980e28039f0f7d6083cf215798f43b639eafeea"}
Oct 30 22:49:03.809865 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.809648    4553 scope.go:122] "RemoveContainer" containerID="beadfa8b8089d06f0ac84a218802270351e6def1a4f37b76f77fc7ee9bce8823"
Oct 30 22:49:03.820524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.820503    4553 scope.go:122] "RemoveContainer" containerID="beadfa8b8089d06f0ac84a218802270351e6def1a4f37b76f77fc7ee9bce8823"
Oct 30 22:49:03.820847 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:49:03.820820    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beadfa8b8089d06f0ac84a218802270351e6def1a4f37b76f77fc7ee9bce8823\": container with ID starting with beadfa8b8089d06f0ac84a218802270351e6def1a4f37b76f77fc7ee9bce8823 not found: ID does not exist" containerID="beadfa8b8089d06f0ac84a218802270351e6def1a4f37b76f77fc7ee9bce8823"
Oct 30 22:49:03.820947 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.820855    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beadfa8b8089d06f0ac84a218802270351e6def1a4f37b76f77fc7ee9bce8823"} err="failed to get container status \"beadfa8b8089d06f0ac84a218802270351e6def1a4f37b76f77fc7ee9bce8823\": rpc error: code = NotFound desc = could not find container \"beadfa8b8089d06f0ac84a218802270351e6def1a4f37b76f77fc7ee9bce8823\": container with ID starting with beadfa8b8089d06f0ac84a218802270351e6def1a4f37b76f77fc7ee9bce8823 not found: ID does not exist"
Oct 30 22:49:03.978264 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.978224    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"]
Oct 30 22:49:03.981306 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:03.981284    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-20ccef-7f969c7869-s77bx"]
Oct 30 22:49:05.395081 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:05.395049    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25abbe77-bfce-472d-84f4-2d5d9bff0c4d" path="/var/lib/kubelet/pods/25abbe77-bfce-472d-84f4-2d5d9bff0c4d/volumes"
Oct 30 22:49:21.403871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:21.403795    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"]
Oct 30 22:49:22.342364 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.342313    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"]
Oct 30 22:49:22.342595 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.342571    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh" podUID="7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5" containerName="maas-api" containerID="cri-o://74938b926a147dce4e1df3b5b206539881272cecb9d3ed646bd194396d99125a" gracePeriod=30
Oct 30 22:49:22.621796 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.621770    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:49:22.691402 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.691373    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-kube-api-access-kbrps\" (UniqueName: \"kubernetes.io/projected/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-kube-api-access-kbrps\") pod \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") "
Oct 30 22:49:22.691620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.691455    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-openshift-service-ca\") pod \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") "
Oct 30 22:49:22.691620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.691477    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-metrics-certs\") pod \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") "
Oct 30 22:49:22.691620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.691593    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-maas-api-tls\") pod \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\" (UID: \"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5\") "
Oct 30 22:49:22.691871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.691847    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-openshift-service-ca" pod "7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5" (UID: "7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:22.691960 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.691939    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:22.693882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.693856    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-metrics-certs" pod "7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5" (UID: "7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:22.693882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.693873    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-kube-api-access-kbrps" pod "7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5" (UID: "7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5"). InnerVolumeSpecName "kube-api-access-kbrps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:22.694045 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.693878    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-maas-api-tls" pod "7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5" (UID: "7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:22.793201 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.793162    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:22.793201 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.793202    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbrps\" (UniqueName: \"kubernetes.io/projected/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-kube-api-access-kbrps\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:22.793418 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.793220    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:22.945810 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.945716    4553 generic.go:356] "Generic (PLEG): container finished" podID="7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5" containerID="74938b926a147dce4e1df3b5b206539881272cecb9d3ed646bd194396d99125a" exitCode=0
Oct 30 22:49:22.945810 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.945765    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"
Oct 30 22:49:22.945810 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.945798    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh" event={"ID":"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5","Type":"ContainerDied","Data":"74938b926a147dce4e1df3b5b206539881272cecb9d3ed646bd194396d99125a"}
Oct 30 22:49:22.946099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.945854    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh" event={"ID":"7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5","Type":"ContainerDied","Data":"82688b0d4c097d470232f2c50a0eb571c47589c8e48447048a8a705f9d962ca0"}
Oct 30 22:49:22.946099 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.945898    4553 scope.go:122] "RemoveContainer" containerID="74938b926a147dce4e1df3b5b206539881272cecb9d3ed646bd194396d99125a"
Oct 30 22:49:22.958020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.958003    4553 scope.go:122] "RemoveContainer" containerID="74938b926a147dce4e1df3b5b206539881272cecb9d3ed646bd194396d99125a"
Oct 30 22:49:22.958359 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:49:22.958338    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74938b926a147dce4e1df3b5b206539881272cecb9d3ed646bd194396d99125a\": container with ID starting with 74938b926a147dce4e1df3b5b206539881272cecb9d3ed646bd194396d99125a not found: ID does not exist" containerID="74938b926a147dce4e1df3b5b206539881272cecb9d3ed646bd194396d99125a"
Oct 30 22:49:22.958426 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:22.958369    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74938b926a147dce4e1df3b5b206539881272cecb9d3ed646bd194396d99125a"} err="failed to get container status \"74938b926a147dce4e1df3b5b206539881272cecb9d3ed646bd194396d99125a\": rpc error: code = NotFound desc = could not find container \"74938b926a147dce4e1df3b5b206539881272cecb9d3ed646bd194396d99125a\": container with ID starting with 74938b926a147dce4e1df3b5b206539881272cecb9d3ed646bd194396d99125a not found: ID does not exist"
Oct 30 22:49:23.128834 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:23.128798    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"]
Oct 30 22:49:23.179247 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:23.179208    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-55edb1-6df75fbf6f-hrjzh"]
Oct 30 22:49:23.394991 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:23.394956    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5" path="/var/lib/kubelet/pods/7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5/volumes"
Oct 30 22:49:26.404451 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:26.404400    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg" podUID="0d13c173-2f05-47f6-837d-4e09f3984114" containerName="payload-pre-processing" containerID="cri-o://2c2f7ca6034ac80b7ca18f1cc5bbc39de3f44ce1d9fdd379537c3b5677f854f1" gracePeriod=25
Oct 30 22:49:26.685344 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:26.685313    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"
Oct 30 22:49:26.765833 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:26.765798    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d13c173-2f05-47f6-837d-4e09f3984114-kube-api-access-whzr2\" (UniqueName: \"kubernetes.io/projected/0d13c173-2f05-47f6-837d-4e09f3984114-kube-api-access-whzr2\") pod \"0d13c173-2f05-47f6-837d-4e09f3984114\" (UID: \"0d13c173-2f05-47f6-837d-4e09f3984114\") "
Oct 30 22:49:26.766049 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:26.765878    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d13c173-2f05-47f6-837d-4e09f3984114-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d13c173-2f05-47f6-837d-4e09f3984114-plugins-config-volume\") pod \"0d13c173-2f05-47f6-837d-4e09f3984114\" (UID: \"0d13c173-2f05-47f6-837d-4e09f3984114\") "
Oct 30 22:49:26.766330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:26.766308    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d13c173-2f05-47f6-837d-4e09f3984114-plugins-config-volume" pod "0d13c173-2f05-47f6-837d-4e09f3984114" (UID: "0d13c173-2f05-47f6-837d-4e09f3984114"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:26.766558 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:26.766538    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d13c173-2f05-47f6-837d-4e09f3984114-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:26.768528 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:26.768501    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d13c173-2f05-47f6-837d-4e09f3984114-kube-api-access-whzr2" pod "0d13c173-2f05-47f6-837d-4e09f3984114" (UID: "0d13c173-2f05-47f6-837d-4e09f3984114"). InnerVolumeSpecName "kube-api-access-whzr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:26.868203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:26.868171    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whzr2\" (UniqueName: \"kubernetes.io/projected/0d13c173-2f05-47f6-837d-4e09f3984114-kube-api-access-whzr2\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:26.974245 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:26.974117    4553 generic.go:356] "Generic (PLEG): container finished" podID="0d13c173-2f05-47f6-837d-4e09f3984114" containerID="2c2f7ca6034ac80b7ca18f1cc5bbc39de3f44ce1d9fdd379537c3b5677f854f1" exitCode=0
Oct 30 22:49:26.974245 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:26.974184    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"
Oct 30 22:49:26.974245 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:26.974184    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg" event={"ID":"0d13c173-2f05-47f6-837d-4e09f3984114","Type":"ContainerDied","Data":"2c2f7ca6034ac80b7ca18f1cc5bbc39de3f44ce1d9fdd379537c3b5677f854f1"}
Oct 30 22:49:26.974541 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:26.974266    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg" event={"ID":"0d13c173-2f05-47f6-837d-4e09f3984114","Type":"ContainerDied","Data":"336dda69df482a1e6be3cf5fc36e323c79309a3d0452dc133792e0b82719f983"}
Oct 30 22:49:26.974541 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:26.974287    4553 scope.go:122] "RemoveContainer" containerID="2c2f7ca6034ac80b7ca18f1cc5bbc39de3f44ce1d9fdd379537c3b5677f854f1"
Oct 30 22:49:26.984936 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:26.984918    4553 scope.go:122] "RemoveContainer" containerID="2c2f7ca6034ac80b7ca18f1cc5bbc39de3f44ce1d9fdd379537c3b5677f854f1"
Oct 30 22:49:26.985227 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:49:26.985210    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2f7ca6034ac80b7ca18f1cc5bbc39de3f44ce1d9fdd379537c3b5677f854f1\": container with ID starting with 2c2f7ca6034ac80b7ca18f1cc5bbc39de3f44ce1d9fdd379537c3b5677f854f1 not found: ID does not exist" containerID="2c2f7ca6034ac80b7ca18f1cc5bbc39de3f44ce1d9fdd379537c3b5677f854f1"
Oct 30 22:49:26.985286 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:26.985234    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2f7ca6034ac80b7ca18f1cc5bbc39de3f44ce1d9fdd379537c3b5677f854f1"} err="failed to get container status \"2c2f7ca6034ac80b7ca18f1cc5bbc39de3f44ce1d9fdd379537c3b5677f854f1\": rpc error: code = NotFound desc = could not find container \"2c2f7ca6034ac80b7ca18f1cc5bbc39de3f44ce1d9fdd379537c3b5677f854f1\": container with ID starting with 2c2f7ca6034ac80b7ca18f1cc5bbc39de3f44ce1d9fdd379537c3b5677f854f1 not found: ID does not exist"
Oct 30 22:49:27.142199 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:27.142163    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"]
Oct 30 22:49:27.147347 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:27.147319    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-9e18b5-7dfc749c75-pr5xg"]
Oct 30 22:49:27.393615 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:27.393581    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d13c173-2f05-47f6-837d-4e09f3984114" path="/var/lib/kubelet/pods/0d13c173-2f05-47f6-837d-4e09f3984114/volumes"
Oct 30 22:49:28.202544 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:49:28.202510    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:49:34.150744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:34.150714    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"]
Oct 30 22:49:34.151183 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:34.150940    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc" podUID="2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf" containerName="maas-api" containerID="cri-o://efb5439848362f662c73a60c9a19a091008d77909a075e913e933b501eaf26c6" gracePeriod=30
Oct 30 22:49:34.432276 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:34.432248    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:49:34.573872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:34.573830    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-maas-api-tls\") pod \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") "
Oct 30 22:49:34.574088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:34.573935    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-kube-api-access-hvr5h\" (UniqueName: \"kubernetes.io/projected/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-kube-api-access-hvr5h\") pod \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") "
Oct 30 22:49:34.574263 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:34.574120    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-metrics-certs\") pod \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") "
Oct 30 22:49:34.574263 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:34.574217    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-openshift-service-ca\") pod \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\" (UID: \"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf\") "
Oct 30 22:49:34.575182 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:34.575114    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-openshift-service-ca" pod "2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf" (UID: "2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:34.577606 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:34.577568    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-kube-api-access-hvr5h" pod "2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf" (UID: "2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf"). InnerVolumeSpecName "kube-api-access-hvr5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:34.578358 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:34.578327    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-maas-api-tls" pod "2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf" (UID: "2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:34.578561 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:34.578537    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-metrics-certs" pod "2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf" (UID: "2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:34.676569 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:34.676472    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:34.676569 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:34.676506    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:34.676569 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:34.676519    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvr5h\" (UniqueName: \"kubernetes.io/projected/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-kube-api-access-hvr5h\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:34.676569 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:34.676534    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:35.035566 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:35.035469    4553 generic.go:356] "Generic (PLEG): container finished" podID="2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf" containerID="efb5439848362f662c73a60c9a19a091008d77909a075e913e933b501eaf26c6" exitCode=0
Oct 30 22:49:35.035566 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:35.035518    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"
Oct 30 22:49:35.035566 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:35.035555    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc" event={"ID":"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf","Type":"ContainerDied","Data":"efb5439848362f662c73a60c9a19a091008d77909a075e913e933b501eaf26c6"}
Oct 30 22:49:35.035853 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:35.035610    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc" event={"ID":"2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf","Type":"ContainerDied","Data":"dc4ab2f9dd1ee2ab61491b06f430f7f28558b2578e68c71138184a84bd6fea4a"}
Oct 30 22:49:35.035853 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:35.035635    4553 scope.go:122] "RemoveContainer" containerID="efb5439848362f662c73a60c9a19a091008d77909a075e913e933b501eaf26c6"
Oct 30 22:49:35.047197 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:35.047116    4553 scope.go:122] "RemoveContainer" containerID="efb5439848362f662c73a60c9a19a091008d77909a075e913e933b501eaf26c6"
Oct 30 22:49:35.047568 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:49:35.047545    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb5439848362f662c73a60c9a19a091008d77909a075e913e933b501eaf26c6\": container with ID starting with efb5439848362f662c73a60c9a19a091008d77909a075e913e933b501eaf26c6 not found: ID does not exist" containerID="efb5439848362f662c73a60c9a19a091008d77909a075e913e933b501eaf26c6"
Oct 30 22:49:35.047642 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:35.047582    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb5439848362f662c73a60c9a19a091008d77909a075e913e933b501eaf26c6"} err="failed to get container status \"efb5439848362f662c73a60c9a19a091008d77909a075e913e933b501eaf26c6\": rpc error: code = NotFound desc = could not find container \"efb5439848362f662c73a60c9a19a091008d77909a075e913e933b501eaf26c6\": container with ID starting with efb5439848362f662c73a60c9a19a091008d77909a075e913e933b501eaf26c6 not found: ID does not exist"
Oct 30 22:49:35.204844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:35.204803    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"]
Oct 30 22:49:35.254791 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:35.254753    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9e18b5-5c88788b5-4bccc"]
Oct 30 22:49:35.394699 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:35.394664    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf" path="/var/lib/kubelet/pods/2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf/volumes"
Oct 30 22:49:45.500385 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:45.500347    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"]
Oct 30 22:49:46.822247 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:46.822207    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"]
Oct 30 22:49:46.822705 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:46.822471    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk" podUID="b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62" containerName="maas-api" containerID="cri-o://caa2ea07cf39f030724500bfc2bcf128c531f31b4879bf9f4049c5e8ff33bf1d" gracePeriod=30
Oct 30 22:49:47.113694 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.113663    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:49:47.123633 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.123585    4553 generic.go:356] "Generic (PLEG): container finished" podID="b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62" containerID="caa2ea07cf39f030724500bfc2bcf128c531f31b4879bf9f4049c5e8ff33bf1d" exitCode=0
Oct 30 22:49:47.123850 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.123668    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk" event={"ID":"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62","Type":"ContainerDied","Data":"caa2ea07cf39f030724500bfc2bcf128c531f31b4879bf9f4049c5e8ff33bf1d"}
Oct 30 22:49:47.123850 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.123772    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk" event={"ID":"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62","Type":"ContainerDied","Data":"4a9578173ad4b365c3d1f9e83536899de7bb08b194a1037d27a9028e5b0a62e4"}
Oct 30 22:49:47.123850 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.123811    4553 scope.go:122] "RemoveContainer" containerID="caa2ea07cf39f030724500bfc2bcf128c531f31b4879bf9f4049c5e8ff33bf1d"
Oct 30 22:49:47.124327 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.124304    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"
Oct 30 22:49:47.136047 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.136018    4553 scope.go:122] "RemoveContainer" containerID="caa2ea07cf39f030724500bfc2bcf128c531f31b4879bf9f4049c5e8ff33bf1d"
Oct 30 22:49:47.136378 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:49:47.136357    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caa2ea07cf39f030724500bfc2bcf128c531f31b4879bf9f4049c5e8ff33bf1d\": container with ID starting with caa2ea07cf39f030724500bfc2bcf128c531f31b4879bf9f4049c5e8ff33bf1d not found: ID does not exist" containerID="caa2ea07cf39f030724500bfc2bcf128c531f31b4879bf9f4049c5e8ff33bf1d"
Oct 30 22:49:47.136441 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.136386    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caa2ea07cf39f030724500bfc2bcf128c531f31b4879bf9f4049c5e8ff33bf1d"} err="failed to get container status \"caa2ea07cf39f030724500bfc2bcf128c531f31b4879bf9f4049c5e8ff33bf1d\": rpc error: code = NotFound desc = could not find container \"caa2ea07cf39f030724500bfc2bcf128c531f31b4879bf9f4049c5e8ff33bf1d\": container with ID starting with caa2ea07cf39f030724500bfc2bcf128c531f31b4879bf9f4049c5e8ff33bf1d not found: ID does not exist"
Oct 30 22:49:47.246922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.246890    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-kube-api-access-wzmnd\" (UniqueName: \"kubernetes.io/projected/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-kube-api-access-wzmnd\") pod \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") "
Oct 30 22:49:47.247108 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.246963    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-maas-api-tls\") pod \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") "
Oct 30 22:49:47.247108 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.246991    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-metrics-certs\") pod \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") "
Oct 30 22:49:47.247273 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.247166    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-openshift-service-ca\") pod \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\" (UID: \"b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62\") "
Oct 30 22:49:47.247804 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.247603    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-openshift-service-ca" pod "b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62" (UID: "b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:47.248040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.248016    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:47.249533 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.249502    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-maas-api-tls" pod "b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62" (UID: "b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:47.249654 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.249575    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-metrics-certs" pod "b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62" (UID: "b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:47.249749 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.249729    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-kube-api-access-wzmnd" pod "b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62" (UID: "b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62"). InnerVolumeSpecName "kube-api-access-wzmnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:47.349770 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.349688    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzmnd\" (UniqueName: \"kubernetes.io/projected/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-kube-api-access-wzmnd\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:47.349770 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.349719    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:47.349770 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.349729    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:47.541471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.541408    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"]
Oct 30 22:49:47.545607 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.545572    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a6900c-66b469b648-lvsvk"]
Oct 30 22:49:47.740857 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.740770    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"]
Oct 30 22:49:47.835748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:47.835715    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"]
Oct 30 22:49:49.400508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:49.397754    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62" path="/var/lib/kubelet/pods/b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62/volumes"
Oct 30 22:49:50.501463 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:50.501402    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g" podUID="0e2cb05a-7549-45c9-9eca-db328f6a07cf" containerName="payload-processing" containerID="cri-o://c457452c915348367b417b9ffd760e5506bb588b9a7b1b2db27e654599ada375" gracePeriod=25
Oct 30 22:49:50.791169 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:50.791120    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"
Oct 30 22:49:50.895972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:50.895938    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e2cb05a-7549-45c9-9eca-db328f6a07cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e2cb05a-7549-45c9-9eca-db328f6a07cf-plugins-config-volume\") pod \"0e2cb05a-7549-45c9-9eca-db328f6a07cf\" (UID: \"0e2cb05a-7549-45c9-9eca-db328f6a07cf\") "
Oct 30 22:49:50.896210 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:50.896188    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e2cb05a-7549-45c9-9eca-db328f6a07cf-kube-api-access-pwtcs\" (UniqueName: \"kubernetes.io/projected/0e2cb05a-7549-45c9-9eca-db328f6a07cf-kube-api-access-pwtcs\") pod \"0e2cb05a-7549-45c9-9eca-db328f6a07cf\" (UID: \"0e2cb05a-7549-45c9-9eca-db328f6a07cf\") "
Oct 30 22:49:50.896445 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:50.896425    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e2cb05a-7549-45c9-9eca-db328f6a07cf-plugins-config-volume" pod "0e2cb05a-7549-45c9-9eca-db328f6a07cf" (UID: "0e2cb05a-7549-45c9-9eca-db328f6a07cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:50.896715 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:50.896687    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e2cb05a-7549-45c9-9eca-db328f6a07cf-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:50.898450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:50.898427    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2cb05a-7549-45c9-9eca-db328f6a07cf-kube-api-access-pwtcs" pod "0e2cb05a-7549-45c9-9eca-db328f6a07cf" (UID: "0e2cb05a-7549-45c9-9eca-db328f6a07cf"). InnerVolumeSpecName "kube-api-access-pwtcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:50.998551 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:50.998509    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwtcs\" (UniqueName: \"kubernetes.io/projected/0e2cb05a-7549-45c9-9eca-db328f6a07cf-kube-api-access-pwtcs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:49:51.152964 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:51.152931    4553 generic.go:356] "Generic (PLEG): container finished" podID="0e2cb05a-7549-45c9-9eca-db328f6a07cf" containerID="c457452c915348367b417b9ffd760e5506bb588b9a7b1b2db27e654599ada375" exitCode=0
Oct 30 22:49:51.153231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:51.153001    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"
Oct 30 22:49:51.153231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:51.153028    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g" event={"ID":"0e2cb05a-7549-45c9-9eca-db328f6a07cf","Type":"ContainerDied","Data":"c457452c915348367b417b9ffd760e5506bb588b9a7b1b2db27e654599ada375"}
Oct 30 22:49:51.153231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:51.153083    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g" event={"ID":"0e2cb05a-7549-45c9-9eca-db328f6a07cf","Type":"ContainerDied","Data":"8f28b38fd26a6cef02078aa8c5673569a44281c8643b604100e0a6d0296ad6a5"}
Oct 30 22:49:51.153231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:51.153108    4553 scope.go:122] "RemoveContainer" containerID="c457452c915348367b417b9ffd760e5506bb588b9a7b1b2db27e654599ada375"
Oct 30 22:49:51.164471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:51.164450    4553 scope.go:122] "RemoveContainer" containerID="c457452c915348367b417b9ffd760e5506bb588b9a7b1b2db27e654599ada375"
Oct 30 22:49:51.164785 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:49:51.164762    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c457452c915348367b417b9ffd760e5506bb588b9a7b1b2db27e654599ada375\": container with ID starting with c457452c915348367b417b9ffd760e5506bb588b9a7b1b2db27e654599ada375 not found: ID does not exist" containerID="c457452c915348367b417b9ffd760e5506bb588b9a7b1b2db27e654599ada375"
Oct 30 22:49:51.164858 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:51.164796    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c457452c915348367b417b9ffd760e5506bb588b9a7b1b2db27e654599ada375"} err="failed to get container status \"c457452c915348367b417b9ffd760e5506bb588b9a7b1b2db27e654599ada375\": rpc error: code = NotFound desc = could not find container \"c457452c915348367b417b9ffd760e5506bb588b9a7b1b2db27e654599ada375\": container with ID starting with c457452c915348367b417b9ffd760e5506bb588b9a7b1b2db27e654599ada375 not found: ID does not exist"
Oct 30 22:49:51.328035 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:51.327994    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"]
Oct 30 22:49:51.332703 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:51.332676    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-15e0f6-6f944c69fc-5927g"]
Oct 30 22:49:51.393979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:51.393945    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e2cb05a-7549-45c9-9eca-db328f6a07cf" path="/var/lib/kubelet/pods/0e2cb05a-7549-45c9-9eca-db328f6a07cf/volumes"
Oct 30 22:49:52.741742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:52.741694    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl" podUID="b4eb8f4c-df16-4758-a3dd-cab73631f335" containerName="payload-processing" containerID="cri-o://eaa124c0435ad9f74018acfc1bb37e71294e9233a1c61d2b7308b2fc05597d60" gracePeriod=25
Oct 30 22:49:52.836766 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:52.836728    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h" podUID="a2fed9e2-b815-4444-b5dc-db25fb1a42c5" containerName="payload-pre-processing" containerID="cri-o://c72ee1a43f69fe736721c968bb299c188411bb9a5a891e9ba0bb5c1a6972647d" gracePeriod=25
Oct 30 22:49:58.203349 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:49:58.203290    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:49:58.503242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:58.503112    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl" podUID="b4eb8f4c-df16-4758-a3dd-cab73631f335" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:49:58.503242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:49:58.503181    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h" podUID="a2fed9e2-b815-4444-b5dc-db25fb1a42c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:50:03.893033 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:03.893000    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"]
Oct 30 22:50:03.893489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:03.893213    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g" podUID="668a3c9f-be16-44ce-91d0-b484530c0d70" containerName="maas-api" containerID="cri-o://56b0b37317cef562f8f72f8b732f55ba2cd42738e60219e2102456f65f25e570" gracePeriod=30
Oct 30 22:50:04.184677 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.184652    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:50:04.244660 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.244627    4553 generic.go:356] "Generic (PLEG): container finished" podID="668a3c9f-be16-44ce-91d0-b484530c0d70" containerID="56b0b37317cef562f8f72f8b732f55ba2cd42738e60219e2102456f65f25e570" exitCode=0
Oct 30 22:50:04.244871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.244683    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"
Oct 30 22:50:04.244871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.244706    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g" event={"ID":"668a3c9f-be16-44ce-91d0-b484530c0d70","Type":"ContainerDied","Data":"56b0b37317cef562f8f72f8b732f55ba2cd42738e60219e2102456f65f25e570"}
Oct 30 22:50:04.244871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.244767    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g" event={"ID":"668a3c9f-be16-44ce-91d0-b484530c0d70","Type":"ContainerDied","Data":"eba92f313ba35adaf0c46be1408699cfe47c5aa92607368e21015bf3f72810ac"}
Oct 30 22:50:04.244871 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.244791    4553 scope.go:122] "RemoveContainer" containerID="56b0b37317cef562f8f72f8b732f55ba2cd42738e60219e2102456f65f25e570"
Oct 30 22:50:04.254831 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.254812    4553 scope.go:122] "RemoveContainer" containerID="56b0b37317cef562f8f72f8b732f55ba2cd42738e60219e2102456f65f25e570"
Oct 30 22:50:04.255121 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:50:04.255099    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b0b37317cef562f8f72f8b732f55ba2cd42738e60219e2102456f65f25e570\": container with ID starting with 56b0b37317cef562f8f72f8b732f55ba2cd42738e60219e2102456f65f25e570 not found: ID does not exist" containerID="56b0b37317cef562f8f72f8b732f55ba2cd42738e60219e2102456f65f25e570"
Oct 30 22:50:04.255246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.255128    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b0b37317cef562f8f72f8b732f55ba2cd42738e60219e2102456f65f25e570"} err="failed to get container status \"56b0b37317cef562f8f72f8b732f55ba2cd42738e60219e2102456f65f25e570\": rpc error: code = NotFound desc = could not find container \"56b0b37317cef562f8f72f8b732f55ba2cd42738e60219e2102456f65f25e570\": container with ID starting with 56b0b37317cef562f8f72f8b732f55ba2cd42738e60219e2102456f65f25e570 not found: ID does not exist"
Oct 30 22:50:04.368693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.368655    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-maas-api-tls\") pod \"668a3c9f-be16-44ce-91d0-b484530c0d70\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") "
Oct 30 22:50:04.368906 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.368717    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668a3c9f-be16-44ce-91d0-b484530c0d70-kube-api-access-pkdtv\" (UniqueName: \"kubernetes.io/projected/668a3c9f-be16-44ce-91d0-b484530c0d70-kube-api-access-pkdtv\") pod \"668a3c9f-be16-44ce-91d0-b484530c0d70\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") "
Oct 30 22:50:04.368906 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.368767    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668a3c9f-be16-44ce-91d0-b484530c0d70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668a3c9f-be16-44ce-91d0-b484530c0d70-openshift-service-ca\") pod \"668a3c9f-be16-44ce-91d0-b484530c0d70\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") "
Oct 30 22:50:04.369038 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.368950    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-metrics-certs\") pod \"668a3c9f-be16-44ce-91d0-b484530c0d70\" (UID: \"668a3c9f-be16-44ce-91d0-b484530c0d70\") "
Oct 30 22:50:04.369241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.369213    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668a3c9f-be16-44ce-91d0-b484530c0d70-openshift-service-ca" pod "668a3c9f-be16-44ce-91d0-b484530c0d70" (UID: "668a3c9f-be16-44ce-91d0-b484530c0d70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:04.369777 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.369755    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/668a3c9f-be16-44ce-91d0-b484530c0d70-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:04.371045 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.371025    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-maas-api-tls" pod "668a3c9f-be16-44ce-91d0-b484530c0d70" (UID: "668a3c9f-be16-44ce-91d0-b484530c0d70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:04.371108 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.371062    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-metrics-certs" pod "668a3c9f-be16-44ce-91d0-b484530c0d70" (UID: "668a3c9f-be16-44ce-91d0-b484530c0d70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:04.371176 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.371164    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668a3c9f-be16-44ce-91d0-b484530c0d70-kube-api-access-pkdtv" pod "668a3c9f-be16-44ce-91d0-b484530c0d70" (UID: "668a3c9f-be16-44ce-91d0-b484530c0d70"). InnerVolumeSpecName "kube-api-access-pkdtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:04.471570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.471430    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:04.471570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.471464    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/668a3c9f-be16-44ce-91d0-b484530c0d70-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:04.471570 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.471481    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkdtv\" (UniqueName: \"kubernetes.io/projected/668a3c9f-be16-44ce-91d0-b484530c0d70-kube-api-access-pkdtv\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:04.668969 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.668931    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"]
Oct 30 22:50:04.675682 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:04.675647    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-20ccef-84f447fb88-csb5g"]
Oct 30 22:50:05.393672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:05.393630    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668a3c9f-be16-44ce-91d0-b484530c0d70" path="/var/lib/kubelet/pods/668a3c9f-be16-44ce-91d0-b484530c0d70/volumes"
Oct 30 22:50:08.427530 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:08.427497    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"]
Oct 30 22:50:08.502655 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:08.502619    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl" podUID="b4eb8f4c-df16-4758-a3dd-cab73631f335" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:50:08.502655 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:08.502649    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h" podUID="a2fed9e2-b815-4444-b5dc-db25fb1a42c5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:50:08.543885 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:08.543851    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"]
Oct 30 22:50:13.172417 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.172390    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"
Oct 30 22:50:13.189909 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.189875    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"
Oct 30 22:50:13.288223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.288084    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2fed9e2-b815-4444-b5dc-db25fb1a42c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fed9e2-b815-4444-b5dc-db25fb1a42c5-plugins-config-volume\") pod \"a2fed9e2-b815-4444-b5dc-db25fb1a42c5\" (UID: \"a2fed9e2-b815-4444-b5dc-db25fb1a42c5\") "
Oct 30 22:50:13.288223 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.288191    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2fed9e2-b815-4444-b5dc-db25fb1a42c5-kube-api-access-btpxg\" (UniqueName: \"kubernetes.io/projected/a2fed9e2-b815-4444-b5dc-db25fb1a42c5-kube-api-access-btpxg\") pod \"a2fed9e2-b815-4444-b5dc-db25fb1a42c5\" (UID: \"a2fed9e2-b815-4444-b5dc-db25fb1a42c5\") "
Oct 30 22:50:13.288513 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.288258    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4eb8f4c-df16-4758-a3dd-cab73631f335-kube-api-access-ck2mg\" (UniqueName: \"kubernetes.io/projected/b4eb8f4c-df16-4758-a3dd-cab73631f335-kube-api-access-ck2mg\") pod \"b4eb8f4c-df16-4758-a3dd-cab73631f335\" (UID: \"b4eb8f4c-df16-4758-a3dd-cab73631f335\") "
Oct 30 22:50:13.288513 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.288359    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4eb8f4c-df16-4758-a3dd-cab73631f335-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4eb8f4c-df16-4758-a3dd-cab73631f335-plugins-config-volume\") pod \"b4eb8f4c-df16-4758-a3dd-cab73631f335\" (UID: \"b4eb8f4c-df16-4758-a3dd-cab73631f335\") "
Oct 30 22:50:13.288941 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.288692    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2fed9e2-b815-4444-b5dc-db25fb1a42c5-plugins-config-volume" pod "a2fed9e2-b815-4444-b5dc-db25fb1a42c5" (UID: "a2fed9e2-b815-4444-b5dc-db25fb1a42c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:13.289279 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.289240    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fed9e2-b815-4444-b5dc-db25fb1a42c5-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:13.289694 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.289601    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4eb8f4c-df16-4758-a3dd-cab73631f335-plugins-config-volume" pod "b4eb8f4c-df16-4758-a3dd-cab73631f335" (UID: "b4eb8f4c-df16-4758-a3dd-cab73631f335"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:13.291119 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.291089    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4eb8f4c-df16-4758-a3dd-cab73631f335-kube-api-access-ck2mg" pod "b4eb8f4c-df16-4758-a3dd-cab73631f335" (UID: "b4eb8f4c-df16-4758-a3dd-cab73631f335"). InnerVolumeSpecName "kube-api-access-ck2mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:13.291428 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.291396    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fed9e2-b815-4444-b5dc-db25fb1a42c5-kube-api-access-btpxg" pod "a2fed9e2-b815-4444-b5dc-db25fb1a42c5" (UID: "a2fed9e2-b815-4444-b5dc-db25fb1a42c5"). InnerVolumeSpecName "kube-api-access-btpxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:13.316466 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.315627    4553 generic.go:356] "Generic (PLEG): container finished" podID="b4eb8f4c-df16-4758-a3dd-cab73631f335" containerID="eaa124c0435ad9f74018acfc1bb37e71294e9233a1c61d2b7308b2fc05597d60" exitCode=0
Oct 30 22:50:13.316466 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.315968    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"
Oct 30 22:50:13.316830 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.316798    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl" event={"ID":"b4eb8f4c-df16-4758-a3dd-cab73631f335","Type":"ContainerDied","Data":"eaa124c0435ad9f74018acfc1bb37e71294e9233a1c61d2b7308b2fc05597d60"}
Oct 30 22:50:13.317080 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.316870    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl" event={"ID":"b4eb8f4c-df16-4758-a3dd-cab73631f335","Type":"ContainerDied","Data":"6292d9043d926c11fc12c2825107504d20938892c8b502ca2d9ce6251b02437d"}
Oct 30 22:50:13.317080 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.316903    4553 scope.go:122] "RemoveContainer" containerID="eaa124c0435ad9f74018acfc1bb37e71294e9233a1c61d2b7308b2fc05597d60"
Oct 30 22:50:13.322734 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.322705    4553 generic.go:356] "Generic (PLEG): container finished" podID="a2fed9e2-b815-4444-b5dc-db25fb1a42c5" containerID="c72ee1a43f69fe736721c968bb299c188411bb9a5a891e9ba0bb5c1a6972647d" exitCode=0
Oct 30 22:50:13.322903 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.322871    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h" event={"ID":"a2fed9e2-b815-4444-b5dc-db25fb1a42c5","Type":"ContainerDied","Data":"c72ee1a43f69fe736721c968bb299c188411bb9a5a891e9ba0bb5c1a6972647d"}
Oct 30 22:50:13.322989 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.322922    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h" event={"ID":"a2fed9e2-b815-4444-b5dc-db25fb1a42c5","Type":"ContainerDied","Data":"05d5c9fb0bb18d816f6d20e3dd22fa7cc6901c6ccde494493b5a427412ee1ecc"}
Oct 30 22:50:13.323056 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.322998    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"
Oct 30 22:50:13.339316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.338515    4553 scope.go:122] "RemoveContainer" containerID="eaa124c0435ad9f74018acfc1bb37e71294e9233a1c61d2b7308b2fc05597d60"
Oct 30 22:50:13.339316 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:50:13.338928    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa124c0435ad9f74018acfc1bb37e71294e9233a1c61d2b7308b2fc05597d60\": container with ID starting with eaa124c0435ad9f74018acfc1bb37e71294e9233a1c61d2b7308b2fc05597d60 not found: ID does not exist" containerID="eaa124c0435ad9f74018acfc1bb37e71294e9233a1c61d2b7308b2fc05597d60"
Oct 30 22:50:13.339316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.338968    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa124c0435ad9f74018acfc1bb37e71294e9233a1c61d2b7308b2fc05597d60"} err="failed to get container status \"eaa124c0435ad9f74018acfc1bb37e71294e9233a1c61d2b7308b2fc05597d60\": rpc error: code = NotFound desc = could not find container \"eaa124c0435ad9f74018acfc1bb37e71294e9233a1c61d2b7308b2fc05597d60\": container with ID starting with eaa124c0435ad9f74018acfc1bb37e71294e9233a1c61d2b7308b2fc05597d60 not found: ID does not exist"
Oct 30 22:50:13.339316 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.338994    4553 scope.go:122] "RemoveContainer" containerID="c72ee1a43f69fe736721c968bb299c188411bb9a5a891e9ba0bb5c1a6972647d"
Oct 30 22:50:13.351002 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.350980    4553 scope.go:122] "RemoveContainer" containerID="c72ee1a43f69fe736721c968bb299c188411bb9a5a891e9ba0bb5c1a6972647d"
Oct 30 22:50:13.351383 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:50:13.351352    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72ee1a43f69fe736721c968bb299c188411bb9a5a891e9ba0bb5c1a6972647d\": container with ID starting with c72ee1a43f69fe736721c968bb299c188411bb9a5a891e9ba0bb5c1a6972647d not found: ID does not exist" containerID="c72ee1a43f69fe736721c968bb299c188411bb9a5a891e9ba0bb5c1a6972647d"
Oct 30 22:50:13.351517 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.351386    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72ee1a43f69fe736721c968bb299c188411bb9a5a891e9ba0bb5c1a6972647d"} err="failed to get container status \"c72ee1a43f69fe736721c968bb299c188411bb9a5a891e9ba0bb5c1a6972647d\": rpc error: code = NotFound desc = could not find container \"c72ee1a43f69fe736721c968bb299c188411bb9a5a891e9ba0bb5c1a6972647d\": container with ID starting with c72ee1a43f69fe736721c968bb299c188411bb9a5a891e9ba0bb5c1a6972647d not found: ID does not exist"
Oct 30 22:50:13.390764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.390735    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4eb8f4c-df16-4758-a3dd-cab73631f335-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:13.390764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.390762    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btpxg\" (UniqueName: \"kubernetes.io/projected/a2fed9e2-b815-4444-b5dc-db25fb1a42c5-kube-api-access-btpxg\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:13.390764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.390773    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck2mg\" (UniqueName: \"kubernetes.io/projected/b4eb8f4c-df16-4758-a3dd-cab73631f335-kube-api-access-ck2mg\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:13.428341 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.428291    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s" podUID="74a81c04-db18-427e-b73d-f4bc5ea33774" containerName="payload-processing" containerID="cri-o://3fb33e986ee6022c27a5506097e57ea43c4a7a031a316cd747e74546f12da409" gracePeriod=25
Oct 30 22:50:13.521491 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.521454    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"]
Oct 30 22:50:13.525608 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.525580    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4deef-7697f6b559-jqh8h"]
Oct 30 22:50:13.546392 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.546359    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv" podUID="1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b" containerName="payload-pre-processing" containerID="cri-o://6d3c0690a47b1cf0d8d04c5a0df9753145a3ef87d83108e04fd03f3448529a9b" gracePeriod=25
Oct 30 22:50:13.641058 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.641021    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"]
Oct 30 22:50:13.646389 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.646298    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4deef-8655ff8547-rbtbl"]
Oct 30 22:50:13.706779 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.706752    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"
Oct 30 22:50:13.797248 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.795591    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a81c04-db18-427e-b73d-f4bc5ea33774-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a81c04-db18-427e-b73d-f4bc5ea33774-plugins-config-volume\") pod \"74a81c04-db18-427e-b73d-f4bc5ea33774\" (UID: \"74a81c04-db18-427e-b73d-f4bc5ea33774\") "
Oct 30 22:50:13.797248 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.795699    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a81c04-db18-427e-b73d-f4bc5ea33774-kube-api-access-ppc8t\" (UniqueName: \"kubernetes.io/projected/74a81c04-db18-427e-b73d-f4bc5ea33774-kube-api-access-ppc8t\") pod \"74a81c04-db18-427e-b73d-f4bc5ea33774\" (UID: \"74a81c04-db18-427e-b73d-f4bc5ea33774\") "
Oct 30 22:50:13.797248 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.796046    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a81c04-db18-427e-b73d-f4bc5ea33774-plugins-config-volume" pod "74a81c04-db18-427e-b73d-f4bc5ea33774" (UID: "74a81c04-db18-427e-b73d-f4bc5ea33774"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:13.797248 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.796492    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a81c04-db18-427e-b73d-f4bc5ea33774-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:13.799500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.799470    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a81c04-db18-427e-b73d-f4bc5ea33774-kube-api-access-ppc8t" pod "74a81c04-db18-427e-b73d-f4bc5ea33774" (UID: "74a81c04-db18-427e-b73d-f4bc5ea33774"). InnerVolumeSpecName "kube-api-access-ppc8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:13.859214 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.859185    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"
Oct 30 22:50:13.897787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.897752    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppc8t\" (UniqueName: \"kubernetes.io/projected/74a81c04-db18-427e-b73d-f4bc5ea33774-kube-api-access-ppc8t\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:13.999007 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.998975    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b-plugins-config-volume\") pod \"1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b\" (UID: \"1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b\") "
Oct 30 22:50:13.999244 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.999029    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b-kube-api-access-sljkk\" (UniqueName: \"kubernetes.io/projected/1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b-kube-api-access-sljkk\") pod \"1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b\" (UID: \"1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b\") "
Oct 30 22:50:13.999474 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:13.999447    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b-plugins-config-volume" pod "1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b" (UID: "1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:14.001472 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.001441    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b-kube-api-access-sljkk" pod "1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b" (UID: "1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b"). InnerVolumeSpecName "kube-api-access-sljkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:14.100210 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.100171    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:14.100210 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.100201    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljkk\" (UniqueName: \"kubernetes.io/projected/1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b-kube-api-access-sljkk\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:14.332025 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.331983    4553 generic.go:356] "Generic (PLEG): container finished" podID="74a81c04-db18-427e-b73d-f4bc5ea33774" containerID="3fb33e986ee6022c27a5506097e57ea43c4a7a031a316cd747e74546f12da409" exitCode=0
Oct 30 22:50:14.332500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.332034    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"
Oct 30 22:50:14.332500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.332068    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s" event={"ID":"74a81c04-db18-427e-b73d-f4bc5ea33774","Type":"ContainerDied","Data":"3fb33e986ee6022c27a5506097e57ea43c4a7a031a316cd747e74546f12da409"}
Oct 30 22:50:14.332500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.332123    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s" event={"ID":"74a81c04-db18-427e-b73d-f4bc5ea33774","Type":"ContainerDied","Data":"8fd4a7404e0cfc8b99b234aa6d818d62dcfdc2040f98f14e1ff5e1975d62ce04"}
Oct 30 22:50:14.332500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.332178    4553 scope.go:122] "RemoveContainer" containerID="3fb33e986ee6022c27a5506097e57ea43c4a7a031a316cd747e74546f12da409"
Oct 30 22:50:14.333616 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.333597    4553 generic.go:356] "Generic (PLEG): container finished" podID="1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b" containerID="6d3c0690a47b1cf0d8d04c5a0df9753145a3ef87d83108e04fd03f3448529a9b" exitCode=0
Oct 30 22:50:14.333701 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.333642    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"
Oct 30 22:50:14.333762 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.333689    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv" event={"ID":"1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b","Type":"ContainerDied","Data":"6d3c0690a47b1cf0d8d04c5a0df9753145a3ef87d83108e04fd03f3448529a9b"}
Oct 30 22:50:14.333890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.333811    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv" event={"ID":"1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b","Type":"ContainerDied","Data":"b9fcaa2dd349105979806952918f712f5e3e0baaad913052ddc0e8c9f30c49d1"}
Oct 30 22:50:14.344945 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.344905    4553 scope.go:122] "RemoveContainer" containerID="3fb33e986ee6022c27a5506097e57ea43c4a7a031a316cd747e74546f12da409"
Oct 30 22:50:14.345263 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:50:14.345225    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb33e986ee6022c27a5506097e57ea43c4a7a031a316cd747e74546f12da409\": container with ID starting with 3fb33e986ee6022c27a5506097e57ea43c4a7a031a316cd747e74546f12da409 not found: ID does not exist" containerID="3fb33e986ee6022c27a5506097e57ea43c4a7a031a316cd747e74546f12da409"
Oct 30 22:50:14.345359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.345273    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb33e986ee6022c27a5506097e57ea43c4a7a031a316cd747e74546f12da409"} err="failed to get container status \"3fb33e986ee6022c27a5506097e57ea43c4a7a031a316cd747e74546f12da409\": rpc error: code = NotFound desc = could not find container \"3fb33e986ee6022c27a5506097e57ea43c4a7a031a316cd747e74546f12da409\": container with ID starting with 3fb33e986ee6022c27a5506097e57ea43c4a7a031a316cd747e74546f12da409 not found: ID does not exist"
Oct 30 22:50:14.345359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.345301    4553 scope.go:122] "RemoveContainer" containerID="6d3c0690a47b1cf0d8d04c5a0df9753145a3ef87d83108e04fd03f3448529a9b"
Oct 30 22:50:14.355978 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.355905    4553 scope.go:122] "RemoveContainer" containerID="6d3c0690a47b1cf0d8d04c5a0df9753145a3ef87d83108e04fd03f3448529a9b"
Oct 30 22:50:14.356272 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:50:14.356244    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d3c0690a47b1cf0d8d04c5a0df9753145a3ef87d83108e04fd03f3448529a9b\": container with ID starting with 6d3c0690a47b1cf0d8d04c5a0df9753145a3ef87d83108e04fd03f3448529a9b not found: ID does not exist" containerID="6d3c0690a47b1cf0d8d04c5a0df9753145a3ef87d83108e04fd03f3448529a9b"
Oct 30 22:50:14.356380 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.356279    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d3c0690a47b1cf0d8d04c5a0df9753145a3ef87d83108e04fd03f3448529a9b"} err="failed to get container status \"6d3c0690a47b1cf0d8d04c5a0df9753145a3ef87d83108e04fd03f3448529a9b\": rpc error: code = NotFound desc = could not find container \"6d3c0690a47b1cf0d8d04c5a0df9753145a3ef87d83108e04fd03f3448529a9b\": container with ID starting with 6d3c0690a47b1cf0d8d04c5a0df9753145a3ef87d83108e04fd03f3448529a9b not found: ID does not exist"
Oct 30 22:50:14.501285 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.501241    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"]
Oct 30 22:50:14.505256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.505230    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-69058c-6f59c784c9-wjgfv"]
Oct 30 22:50:14.615682 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.615578    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"]
Oct 30 22:50:14.619011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:14.618985    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69058c-7b4b756787-tw68s"]
Oct 30 22:50:15.393922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:15.393890    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b" path="/var/lib/kubelet/pods/1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b/volumes"
Oct 30 22:50:15.394357 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:15.394335    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a81c04-db18-427e-b73d-f4bc5ea33774" path="/var/lib/kubelet/pods/74a81c04-db18-427e-b73d-f4bc5ea33774/volumes"
Oct 30 22:50:15.394688 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:15.394675    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2fed9e2-b815-4444-b5dc-db25fb1a42c5" path="/var/lib/kubelet/pods/a2fed9e2-b815-4444-b5dc-db25fb1a42c5/volumes"
Oct 30 22:50:15.395038 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:15.395026    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4eb8f4c-df16-4758-a3dd-cab73631f335" path="/var/lib/kubelet/pods/b4eb8f4c-df16-4758-a3dd-cab73631f335/volumes"
Oct 30 22:50:18.104447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.104414    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"]
Oct 30 22:50:18.105280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105256    4553 cpu_manager.go:414] "containerMap: removing container" podUID="25abbe77-bfce-472d-84f4-2d5d9bff0c4d" containerName="payload-processing"
Oct 30 22:50:18.105280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105279    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25abbe77-bfce-472d-84f4-2d5d9bff0c4d" containerName="payload-processing"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105292    4553 cpu_manager.go:414] "containerMap: removing container" podUID="0e2cb05a-7549-45c9-9eca-db328f6a07cf" containerName="payload-processing"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105300    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2cb05a-7549-45c9-9eca-db328f6a07cf" containerName="payload-processing"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105325    4553 cpu_manager.go:414] "containerMap: removing container" podUID="843153d0-0973-45a5-b4bc-063f6ea52968" containerName="payload-pre-processing"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105333    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="843153d0-0973-45a5-b4bc-063f6ea52968" containerName="payload-pre-processing"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105362    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5" containerName="maas-api"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105369    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5" containerName="maas-api"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105383    4553 cpu_manager.go:414] "containerMap: removing container" podUID="1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b" containerName="payload-pre-processing"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105390    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b" containerName="payload-pre-processing"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105407    4553 cpu_manager.go:414] "containerMap: removing container" podUID="0d13c173-2f05-47f6-837d-4e09f3984114" containerName="payload-pre-processing"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105415    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d13c173-2f05-47f6-837d-4e09f3984114" containerName="payload-pre-processing"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105432    4553 cpu_manager.go:414] "containerMap: removing container" podUID="b4eb8f4c-df16-4758-a3dd-cab73631f335" containerName="payload-processing"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105439    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4eb8f4c-df16-4758-a3dd-cab73631f335" containerName="payload-processing"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105452    4553 cpu_manager.go:414] "containerMap: removing container" podUID="668a3c9f-be16-44ce-91d0-b484530c0d70" containerName="maas-api"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105459    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668a3c9f-be16-44ce-91d0-b484530c0d70" containerName="maas-api"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105471    4553 cpu_manager.go:414] "containerMap: removing container" podUID="b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62" containerName="maas-api"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105477    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62" containerName="maas-api"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105486    4553 cpu_manager.go:414] "containerMap: removing container" podUID="74a81c04-db18-427e-b73d-f4bc5ea33774" containerName="payload-processing"
Oct 30 22:50:18.105489 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105493    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a81c04-db18-427e-b73d-f4bc5ea33774" containerName="payload-processing"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105503    4553 cpu_manager.go:414] "containerMap: removing container" podUID="2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf" containerName="maas-api"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105509    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf" containerName="maas-api"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105530    4553 cpu_manager.go:414] "containerMap: removing container" podUID="a2fed9e2-b815-4444-b5dc-db25fb1a42c5" containerName="payload-pre-processing"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105537    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2fed9e2-b815-4444-b5dc-db25fb1a42c5" containerName="payload-pre-processing"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105686    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7453d4cd-7f1a-42e0-a950-0fb3ca01aaf5" containerName="maas-api"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105699    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aaa183a-c1f4-4b5f-aecb-5f11e4575c1b" containerName="payload-pre-processing"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105712    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d13c173-2f05-47f6-837d-4e09f3984114" containerName="payload-pre-processing"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105723    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4eb8f4c-df16-4758-a3dd-cab73631f335" containerName="payload-processing"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105731    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="668a3c9f-be16-44ce-91d0-b484530c0d70" containerName="maas-api"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105740    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d2b8a7-f082-4a3a-adf7-ff2de7b9bb62" containerName="maas-api"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105748    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a81c04-db18-427e-b73d-f4bc5ea33774" containerName="payload-processing"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105757    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="2715ecd9-7cfe-448f-9cd6-0899e0e8e5bf" containerName="maas-api"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105766    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2fed9e2-b815-4444-b5dc-db25fb1a42c5" containerName="payload-pre-processing"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105780    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="25abbe77-bfce-472d-84f4-2d5d9bff0c4d" containerName="payload-processing"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105791    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e2cb05a-7549-45c9-9eca-db328f6a07cf" containerName="payload-processing"
Oct 30 22:50:18.106434 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.105800    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="843153d0-0973-45a5-b4bc-063f6ea52968" containerName="payload-pre-processing"
Oct 30 22:50:18.111218 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.111118    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"
Oct 30 22:50:18.169819 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.169210    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"]
Oct 30 22:50:18.219206 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.212403    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-6924e5\""
Oct 30 22:50:18.219206 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.212550    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-6924e5-dockercfg-mcdfg\""
Oct 30 22:50:18.255773 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.255743    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrf2w\" (UniqueName: \"kubernetes.io/projected/e2b08777-7389-4a3c-acc8-95e0c5a274bd-kube-api-access-mrf2w\") pod \"payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr\" (UID: \"e2b08777-7389-4a3c-acc8-95e0c5a274bd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"
Oct 30 22:50:18.255956 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.255885    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2b08777-7389-4a3c-acc8-95e0c5a274bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr\" (UID: \"e2b08777-7389-4a3c-acc8-95e0c5a274bd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"
Oct 30 22:50:18.266006 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.265983    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"]
Oct 30 22:50:18.270329 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.270313    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"
Oct 30 22:50:18.312181 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.312148    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"]
Oct 30 22:50:18.357416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.357324    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2b08777-7389-4a3c-acc8-95e0c5a274bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr\" (UID: \"e2b08777-7389-4a3c-acc8-95e0c5a274bd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"
Oct 30 22:50:18.357416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.357394    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsmzk\" (UniqueName: \"kubernetes.io/projected/7103581d-3919-4cb9-bae3-9ade62f0ed1e-kube-api-access-lsmzk\") pod \"payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s\" (UID: \"7103581d-3919-4cb9-bae3-9ade62f0ed1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"
Oct 30 22:50:18.357653 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.357535    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7103581d-3919-4cb9-bae3-9ade62f0ed1e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s\" (UID: \"7103581d-3919-4cb9-bae3-9ade62f0ed1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"
Oct 30 22:50:18.357653 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.357640    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrf2w\" (UniqueName: \"kubernetes.io/projected/e2b08777-7389-4a3c-acc8-95e0c5a274bd-kube-api-access-mrf2w\") pod \"payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr\" (UID: \"e2b08777-7389-4a3c-acc8-95e0c5a274bd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"
Oct 30 22:50:18.358101 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.358068    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2b08777-7389-4a3c-acc8-95e0c5a274bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr\" (UID: \"e2b08777-7389-4a3c-acc8-95e0c5a274bd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"
Oct 30 22:50:18.415746 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.415718    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrf2w\" (UniqueName: \"kubernetes.io/projected/e2b08777-7389-4a3c-acc8-95e0c5a274bd-kube-api-access-mrf2w\") pod \"payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr\" (UID: \"e2b08777-7389-4a3c-acc8-95e0c5a274bd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"
Oct 30 22:50:18.425986 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.425964    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"
Oct 30 22:50:18.458676 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.458638    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7103581d-3919-4cb9-bae3-9ade62f0ed1e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s\" (UID: \"7103581d-3919-4cb9-bae3-9ade62f0ed1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"
Oct 30 22:50:18.458864 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.458810    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsmzk\" (UniqueName: \"kubernetes.io/projected/7103581d-3919-4cb9-bae3-9ade62f0ed1e-kube-api-access-lsmzk\") pod \"payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s\" (UID: \"7103581d-3919-4cb9-bae3-9ade62f0ed1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"
Oct 30 22:50:18.459445 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.459417    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7103581d-3919-4cb9-bae3-9ade62f0ed1e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s\" (UID: \"7103581d-3919-4cb9-bae3-9ade62f0ed1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"
Oct 30 22:50:18.516549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.516519    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsmzk\" (UniqueName: \"kubernetes.io/projected/7103581d-3919-4cb9-bae3-9ade62f0ed1e-kube-api-access-lsmzk\") pod \"payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s\" (UID: \"7103581d-3919-4cb9-bae3-9ade62f0ed1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"
Oct 30 22:50:18.583865 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.583833    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"
Oct 30 22:50:18.806808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.806775    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"]
Oct 30 22:50:18.813898 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:50:18.813871    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b08777_7389_4a3c_acc8_95e0c5a274bd.slice/crio-b00dc448ca471f3ad8c6f4139834e4cb9a5ee231229bc3a16bc7cf3cd29bb30d WatchSource:0}: Error finding container b00dc448ca471f3ad8c6f4139834e4cb9a5ee231229bc3a16bc7cf3cd29bb30d: Status 404 returned error can't find the container with id b00dc448ca471f3ad8c6f4139834e4cb9a5ee231229bc3a16bc7cf3cd29bb30d
Oct 30 22:50:18.965226 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:18.965199    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"]
Oct 30 22:50:18.966148 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:50:18.966111    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7103581d_3919_4cb9_bae3_9ade62f0ed1e.slice/crio-e60de3cd334086a60b5672e24fce2bdaee7fe388b836f2060be4bb6fef86765c WatchSource:0}: Error finding container e60de3cd334086a60b5672e24fce2bdaee7fe388b836f2060be4bb6fef86765c: Status 404 returned error can't find the container with id e60de3cd334086a60b5672e24fce2bdaee7fe388b836f2060be4bb6fef86765c
Oct 30 22:50:19.393921 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:19.393879    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"
Oct 30 22:50:19.394394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:19.393939    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s" event={"ID":"7103581d-3919-4cb9-bae3-9ade62f0ed1e","Type":"ContainerStarted","Data":"be0bde7b47f4949ec4b1d1021cc0e051e5cf696774b47f1390e6a3d2a5034840"}
Oct 30 22:50:19.394394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:19.393964    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s" event={"ID":"7103581d-3919-4cb9-bae3-9ade62f0ed1e","Type":"ContainerStarted","Data":"e60de3cd334086a60b5672e24fce2bdaee7fe388b836f2060be4bb6fef86765c"}
Oct 30 22:50:19.394394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:19.393986    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"
Oct 30 22:50:19.394394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:19.394016    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr" event={"ID":"e2b08777-7389-4a3c-acc8-95e0c5a274bd","Type":"ContainerStarted","Data":"2eda2b2c16c6e2ab95ee60e3ce9b3f1353314cf5fde035969891790a26845f08"}
Oct 30 22:50:19.394394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:19.394037    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr" event={"ID":"e2b08777-7389-4a3c-acc8-95e0c5a274bd","Type":"ContainerStarted","Data":"b00dc448ca471f3ad8c6f4139834e4cb9a5ee231229bc3a16bc7cf3cd29bb30d"}
Oct 30 22:50:19.505979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:19.505915    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s" podStartSLOduration=1.505902088 podStartE2EDuration="1.505902088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:19.503899387 +0000 UTC m=+4070.238262418" watchObservedRunningTime="2026-10-30 22:50:19.505902088 +0000 UTC m=+4070.240265118"
Oct 30 22:50:28.204402 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:50:28.204361    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:50:30.401650 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:30.401616    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"
Oct 30 22:50:30.519668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:30.519613    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr" podStartSLOduration=12.519598258 podStartE2EDuration="12.519598258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:19.62338825 +0000 UTC m=+4070.357751292" watchObservedRunningTime="2026-10-30 22:50:30.519598258 +0000 UTC m=+4081.253961319"
Oct 30 22:50:31.408229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:31.408200    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"
Oct 30 22:50:39.948149 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:39.948106    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"]
Oct 30 22:50:39.948643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:39.948364    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm" podUID="226981d5-7cad-4f8b-9a87-e3c6bef2e59a" containerName="maas-api" containerID="cri-o://b14296b50f51eaef29063890dd88dc505ecac7ee335c4af85f6ae83bdd181cf2" gracePeriod=30
Oct 30 22:50:40.215241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.215174    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:50:40.346738 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.346683    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-openshift-service-ca\") pod \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") "
Oct 30 22:50:40.346962 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.346814    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-maas-api-tls\") pod \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") "
Oct 30 22:50:40.346962 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.346850    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-metrics-certs\") pod \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") "
Oct 30 22:50:40.346962 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.346876    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-kube-api-access-7s2gz\" (UniqueName: \"kubernetes.io/projected/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-kube-api-access-7s2gz\") pod \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\" (UID: \"226981d5-7cad-4f8b-9a87-e3c6bef2e59a\") "
Oct 30 22:50:40.351823 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.351765    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-maas-api-tls" pod "226981d5-7cad-4f8b-9a87-e3c6bef2e59a" (UID: "226981d5-7cad-4f8b-9a87-e3c6bef2e59a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:40.352321 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.352295    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-metrics-certs" pod "226981d5-7cad-4f8b-9a87-e3c6bef2e59a" (UID: "226981d5-7cad-4f8b-9a87-e3c6bef2e59a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:40.352528 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.352507    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-kube-api-access-7s2gz" pod "226981d5-7cad-4f8b-9a87-e3c6bef2e59a" (UID: "226981d5-7cad-4f8b-9a87-e3c6bef2e59a"). InnerVolumeSpecName "kube-api-access-7s2gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:40.356390 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.356352    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-openshift-service-ca" pod "226981d5-7cad-4f8b-9a87-e3c6bef2e59a" (UID: "226981d5-7cad-4f8b-9a87-e3c6bef2e59a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:40.450191 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.450155    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:40.450485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.450458    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:40.450485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.450489    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:40.450758 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.450506    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s2gz\" (UniqueName: \"kubernetes.io/projected/226981d5-7cad-4f8b-9a87-e3c6bef2e59a-kube-api-access-7s2gz\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:40.531958 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.531856    4553 generic.go:356] "Generic (PLEG): container finished" podID="226981d5-7cad-4f8b-9a87-e3c6bef2e59a" containerID="b14296b50f51eaef29063890dd88dc505ecac7ee335c4af85f6ae83bdd181cf2" exitCode=0
Oct 30 22:50:40.532222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.531991    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm" event={"ID":"226981d5-7cad-4f8b-9a87-e3c6bef2e59a","Type":"ContainerDied","Data":"b14296b50f51eaef29063890dd88dc505ecac7ee335c4af85f6ae83bdd181cf2"}
Oct 30 22:50:40.532222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.532078    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm" event={"ID":"226981d5-7cad-4f8b-9a87-e3c6bef2e59a","Type":"ContainerDied","Data":"622de89e36c1dd24b2e7900f65fe60294019ff77c2adc246ddc78c3fe1fbfdee"}
Oct 30 22:50:40.532222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.532116    4553 scope.go:122] "RemoveContainer" containerID="b14296b50f51eaef29063890dd88dc505ecac7ee335c4af85f6ae83bdd181cf2"
Oct 30 22:50:40.532456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.532257    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"
Oct 30 22:50:40.544619 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.544599    4553 scope.go:122] "RemoveContainer" containerID="b14296b50f51eaef29063890dd88dc505ecac7ee335c4af85f6ae83bdd181cf2"
Oct 30 22:50:40.544945 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:50:40.544925    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b14296b50f51eaef29063890dd88dc505ecac7ee335c4af85f6ae83bdd181cf2\": container with ID starting with b14296b50f51eaef29063890dd88dc505ecac7ee335c4af85f6ae83bdd181cf2 not found: ID does not exist" containerID="b14296b50f51eaef29063890dd88dc505ecac7ee335c4af85f6ae83bdd181cf2"
Oct 30 22:50:40.545019 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.544959    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b14296b50f51eaef29063890dd88dc505ecac7ee335c4af85f6ae83bdd181cf2"} err="failed to get container status \"b14296b50f51eaef29063890dd88dc505ecac7ee335c4af85f6ae83bdd181cf2\": rpc error: code = NotFound desc = could not find container \"b14296b50f51eaef29063890dd88dc505ecac7ee335c4af85f6ae83bdd181cf2\": container with ID starting with b14296b50f51eaef29063890dd88dc505ecac7ee335c4af85f6ae83bdd181cf2 not found: ID does not exist"
Oct 30 22:50:40.700554 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.700499    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"]
Oct 30 22:50:40.704250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:40.704225    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-32af64-769d8947d8-nvkxm"]
Oct 30 22:50:41.394818 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:41.394773    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="226981d5-7cad-4f8b-9a87-e3c6bef2e59a" path="/var/lib/kubelet/pods/226981d5-7cad-4f8b-9a87-e3c6bef2e59a/volumes"
Oct 30 22:50:46.227742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.227703    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"]
Oct 30 22:50:46.228275 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.227924    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6" podUID="b2d352d2-8937-4ae7-9c60-2b5324fcea80" containerName="maas-api" containerID="cri-o://2261368e151727a60e9ca2251ab0374172588d6c5880274ceb6e0cc596c1e607" gracePeriod=30
Oct 30 22:50:46.528464 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.528430    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:50:46.572405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.572375    4553 generic.go:356] "Generic (PLEG): container finished" podID="b2d352d2-8937-4ae7-9c60-2b5324fcea80" containerID="2261368e151727a60e9ca2251ab0374172588d6c5880274ceb6e0cc596c1e607" exitCode=0
Oct 30 22:50:46.572671 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.572431    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"
Oct 30 22:50:46.572671 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.572469    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6" event={"ID":"b2d352d2-8937-4ae7-9c60-2b5324fcea80","Type":"ContainerDied","Data":"2261368e151727a60e9ca2251ab0374172588d6c5880274ceb6e0cc596c1e607"}
Oct 30 22:50:46.572671 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.572533    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6" event={"ID":"b2d352d2-8937-4ae7-9c60-2b5324fcea80","Type":"ContainerDied","Data":"71c115be4375fab669a6300c192cb6e46e657b276673c755117722fdae7d6f50"}
Oct 30 22:50:46.572671 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.572562    4553 scope.go:122] "RemoveContainer" containerID="2261368e151727a60e9ca2251ab0374172588d6c5880274ceb6e0cc596c1e607"
Oct 30 22:50:46.583264 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.583245    4553 scope.go:122] "RemoveContainer" containerID="2261368e151727a60e9ca2251ab0374172588d6c5880274ceb6e0cc596c1e607"
Oct 30 22:50:46.583583 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:50:46.583560    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2261368e151727a60e9ca2251ab0374172588d6c5880274ceb6e0cc596c1e607\": container with ID starting with 2261368e151727a60e9ca2251ab0374172588d6c5880274ceb6e0cc596c1e607 not found: ID does not exist" containerID="2261368e151727a60e9ca2251ab0374172588d6c5880274ceb6e0cc596c1e607"
Oct 30 22:50:46.583637 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.583594    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2261368e151727a60e9ca2251ab0374172588d6c5880274ceb6e0cc596c1e607"} err="failed to get container status \"2261368e151727a60e9ca2251ab0374172588d6c5880274ceb6e0cc596c1e607\": rpc error: code = NotFound desc = could not find container \"2261368e151727a60e9ca2251ab0374172588d6c5880274ceb6e0cc596c1e607\": container with ID starting with 2261368e151727a60e9ca2251ab0374172588d6c5880274ceb6e0cc596c1e607 not found: ID does not exist"
Oct 30 22:50:46.623789 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.623758    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-maas-api-tls\") pod \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") "
Oct 30 22:50:46.623970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.623893    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2d352d2-8937-4ae7-9c60-2b5324fcea80-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d352d2-8937-4ae7-9c60-2b5324fcea80-openshift-service-ca\") pod \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") "
Oct 30 22:50:46.623970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.623926    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-metrics-certs\") pod \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") "
Oct 30 22:50:46.623970 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.623952    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d352d2-8937-4ae7-9c60-2b5324fcea80-kube-api-access-p6c6h\" (UniqueName: \"kubernetes.io/projected/b2d352d2-8937-4ae7-9c60-2b5324fcea80-kube-api-access-p6c6h\") pod \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\" (UID: \"b2d352d2-8937-4ae7-9c60-2b5324fcea80\") "
Oct 30 22:50:46.624317 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.624290    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d352d2-8937-4ae7-9c60-2b5324fcea80-openshift-service-ca" pod "b2d352d2-8937-4ae7-9c60-2b5324fcea80" (UID: "b2d352d2-8937-4ae7-9c60-2b5324fcea80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:46.624574 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.624503    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2d352d2-8937-4ae7-9c60-2b5324fcea80-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.626120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.626097    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-maas-api-tls" pod "b2d352d2-8937-4ae7-9c60-2b5324fcea80" (UID: "b2d352d2-8937-4ae7-9c60-2b5324fcea80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:46.626255 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.626233    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d352d2-8937-4ae7-9c60-2b5324fcea80-kube-api-access-p6c6h" pod "b2d352d2-8937-4ae7-9c60-2b5324fcea80" (UID: "b2d352d2-8937-4ae7-9c60-2b5324fcea80"). InnerVolumeSpecName "kube-api-access-p6c6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:46.626255 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.626244    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-metrics-certs" pod "b2d352d2-8937-4ae7-9c60-2b5324fcea80" (UID: "b2d352d2-8937-4ae7-9c60-2b5324fcea80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:46.725725 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.725686    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.725725 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.725723    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6c6h\" (UniqueName: \"kubernetes.io/projected/b2d352d2-8937-4ae7-9c60-2b5324fcea80-kube-api-access-p6c6h\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.725725 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:46.725735    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2d352d2-8937-4ae7-9c60-2b5324fcea80-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:47.001784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:47.001745    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"]
Oct 30 22:50:47.007559 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:47.007531    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b4deef-55bb8db64b-6stc6"]
Oct 30 22:50:47.394669 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:47.394635    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2d352d2-8937-4ae7-9c60-2b5324fcea80" path="/var/lib/kubelet/pods/b2d352d2-8937-4ae7-9c60-2b5324fcea80/volumes"
Oct 30 22:50:49.029040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:49.029004    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"]
Oct 30 22:50:49.029562 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:49.029232    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerName="istio-proxy" containerID="cri-o://3fbda9fb903695a0a59557e5484a63289534070470c8870001aba1d20aaafa3d" gracePeriod=30
Oct 30 22:50:52.257681 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:52.257638    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:50:52.258061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:52.257716    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:50:54.309817 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.309751    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:50:54.420340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.420297    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-podinfo\") pod \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") "
Oct 30 22:50:54.420558 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.420356    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-workload-socket\") pod \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") "
Oct 30 22:50:54.420558 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.420389    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istiod-ca-cert\") pod \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") "
Oct 30 22:50:54.420558 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.420528    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-credential-socket\") pod \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") "
Oct 30 22:50:54.420740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.420647    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-token\" (UniqueName: \"kubernetes.io/projected/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-token\") pod \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") "
Oct 30 22:50:54.420740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.420733    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-envoy\") pod \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") "
Oct 30 22:50:54.420860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.420781    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-data\") pod \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") "
Oct 30 22:50:54.420860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.420732    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-workload-socket" pod "d5f83e3e-757f-495d-9734-e3f7a77d9b9d" (UID: "d5f83e3e-757f-495d-9734-e3f7a77d9b9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:54.420968 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.420787    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-credential-socket" pod "d5f83e3e-757f-495d-9734-e3f7a77d9b9d" (UID: "d5f83e3e-757f-495d-9734-e3f7a77d9b9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:54.420968 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.420888    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-workload-certs\") pod \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") "
Oct 30 22:50:54.421075 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.420974    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-data" pod "d5f83e3e-757f-495d-9734-e3f7a77d9b9d" (UID: "d5f83e3e-757f-495d-9734-e3f7a77d9b9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:54.421075 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.421024    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-kube-api-access-ncpgb\" (UniqueName: \"kubernetes.io/projected/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-kube-api-access-ncpgb\") pod \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\" (UID: \"d5f83e3e-757f-495d-9734-e3f7a77d9b9d\") "
Oct 30 22:50:54.421075 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.421052    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-workload-certs" pod "d5f83e3e-757f-495d-9734-e3f7a77d9b9d" (UID: "d5f83e3e-757f-495d-9734-e3f7a77d9b9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:54.421287 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.421203    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istiod-ca-cert" pod "d5f83e3e-757f-495d-9734-e3f7a77d9b9d" (UID: "d5f83e3e-757f-495d-9734-e3f7a77d9b9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:54.421541 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.421524    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:54.421618 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.421548    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:54.421618 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.421563    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:54.421618 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.421576    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:54.421618 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.421590    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:54.422891 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.422869    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-podinfo" pod "d5f83e3e-757f-495d-9734-e3f7a77d9b9d" (UID: "d5f83e3e-757f-495d-9734-e3f7a77d9b9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:50:54.423212 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.423193    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-kube-api-access-ncpgb" pod "d5f83e3e-757f-495d-9734-e3f7a77d9b9d" (UID: "d5f83e3e-757f-495d-9734-e3f7a77d9b9d"). InnerVolumeSpecName "kube-api-access-ncpgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:54.423416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.423399    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-envoy" pod "d5f83e3e-757f-495d-9734-e3f7a77d9b9d" (UID: "d5f83e3e-757f-495d-9734-e3f7a77d9b9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:54.423482 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.423422    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-token" pod "d5f83e3e-757f-495d-9734-e3f7a77d9b9d" (UID: "d5f83e3e-757f-495d-9734-e3f7a77d9b9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:54.523563 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.523524    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:54.523563 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.523563    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:54.523804 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.523578    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncpgb\" (UniqueName: \"kubernetes.io/projected/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-kube-api-access-ncpgb\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:54.523804 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.523591    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5f83e3e-757f-495d-9734-e3f7a77d9b9d-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:54.626195 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.626158    4553 generic.go:356] "Generic (PLEG): container finished" podID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerID="3fbda9fb903695a0a59557e5484a63289534070470c8870001aba1d20aaafa3d" exitCode=0
Oct 30 22:50:54.626365 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.626214    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"
Oct 30 22:50:54.626365 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.626251    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" event={"ID":"d5f83e3e-757f-495d-9734-e3f7a77d9b9d","Type":"ContainerDied","Data":"3fbda9fb903695a0a59557e5484a63289534070470c8870001aba1d20aaafa3d"}
Oct 30 22:50:54.626365 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.626311    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk" event={"ID":"d5f83e3e-757f-495d-9734-e3f7a77d9b9d","Type":"ContainerDied","Data":"0181e7aec2089794c965d3c1bc41bf8219fcb219695487f609659f6d75c1288c"}
Oct 30 22:50:54.626365 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.626334    4553 scope.go:122] "RemoveContainer" containerID="3fbda9fb903695a0a59557e5484a63289534070470c8870001aba1d20aaafa3d"
Oct 30 22:50:54.638044 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.638020    4553 scope.go:122] "RemoveContainer" containerID="3e931b688cc5dd3cde0f474d1373e89336409217d59519d2bdd4ecdcf49ffd1d"
Oct 30 22:50:54.651702 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.651679    4553 scope.go:122] "RemoveContainer" containerID="3fbda9fb903695a0a59557e5484a63289534070470c8870001aba1d20aaafa3d"
Oct 30 22:50:54.652050 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:50:54.652032    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fbda9fb903695a0a59557e5484a63289534070470c8870001aba1d20aaafa3d\": container with ID starting with 3fbda9fb903695a0a59557e5484a63289534070470c8870001aba1d20aaafa3d not found: ID does not exist" containerID="3fbda9fb903695a0a59557e5484a63289534070470c8870001aba1d20aaafa3d"
Oct 30 22:50:54.652109 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.652060    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fbda9fb903695a0a59557e5484a63289534070470c8870001aba1d20aaafa3d"} err="failed to get container status \"3fbda9fb903695a0a59557e5484a63289534070470c8870001aba1d20aaafa3d\": rpc error: code = NotFound desc = could not find container \"3fbda9fb903695a0a59557e5484a63289534070470c8870001aba1d20aaafa3d\": container with ID starting with 3fbda9fb903695a0a59557e5484a63289534070470c8870001aba1d20aaafa3d not found: ID does not exist"
Oct 30 22:50:54.652109 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.652079    4553 scope.go:122] "RemoveContainer" containerID="3e931b688cc5dd3cde0f474d1373e89336409217d59519d2bdd4ecdcf49ffd1d"
Oct 30 22:50:54.652463 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:50:54.652446    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e931b688cc5dd3cde0f474d1373e89336409217d59519d2bdd4ecdcf49ffd1d\": container with ID starting with 3e931b688cc5dd3cde0f474d1373e89336409217d59519d2bdd4ecdcf49ffd1d not found: ID does not exist" containerID="3e931b688cc5dd3cde0f474d1373e89336409217d59519d2bdd4ecdcf49ffd1d"
Oct 30 22:50:54.652517 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.652466    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e931b688cc5dd3cde0f474d1373e89336409217d59519d2bdd4ecdcf49ffd1d"} err="failed to get container status \"3e931b688cc5dd3cde0f474d1373e89336409217d59519d2bdd4ecdcf49ffd1d\": rpc error: code = NotFound desc = could not find container \"3e931b688cc5dd3cde0f474d1373e89336409217d59519d2bdd4ecdcf49ffd1d\": container with ID starting with 3e931b688cc5dd3cde0f474d1373e89336409217d59519d2bdd4ecdcf49ffd1d not found: ID does not exist"
Oct 30 22:50:54.800611 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.800577    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"]
Oct 30 22:50:54.805116 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:54.805090    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-32af64-openshift-default-8d46c9697-cs2vk"]
Oct 30 22:50:55.394118 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:55.394080    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" path="/var/lib/kubelet/pods/d5f83e3e-757f-495d-9734-e3f7a77d9b9d/volumes"
Oct 30 22:50:55.690683 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:55.690588    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"]
Oct 30 22:50:55.690898 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:55.690859    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp" podUID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerName="istio-proxy" containerID="cri-o://abb7db58614a51bfd8b6343b285d02e6506c28aff6c399c0b56fb6751db9cb1b" gracePeriod=30
Oct 30 22:50:58.202752 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:50:58.202715    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:50:59.398147 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:59.398089    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:50:59.398579 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:50:59.398176    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp" podUID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:51:00.962107 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:00.962034    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:51:01.008213 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.008173    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-token\" (UniqueName: \"kubernetes.io/projected/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-token\") pod \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") "
Oct 30 22:51:01.008393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.008227    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istiod-ca-cert\") pod \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") "
Oct 30 22:51:01.008393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.008291    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e1ba27-d171-44ac-b625-2d2e34fb1481-kube-api-access-4fbwf\" (UniqueName: \"kubernetes.io/projected/c6e1ba27-d171-44ac-b625-2d2e34fb1481-kube-api-access-4fbwf\") pod \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") "
Oct 30 22:51:01.008393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.008320    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-data\") pod \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") "
Oct 30 22:51:01.008393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.008380    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-credential-socket\") pod \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") "
Oct 30 22:51:01.008584 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.008455    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-workload-socket\") pod \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") "
Oct 30 22:51:01.008641 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.008582    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-podinfo\") pod \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") "
Oct 30 22:51:01.008698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.008658    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-data" pod "c6e1ba27-d171-44ac-b625-2d2e34fb1481" (UID: "c6e1ba27-d171-44ac-b625-2d2e34fb1481"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:01.008698 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.008687    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-envoy\") pod \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") "
Oct 30 22:51:01.008802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.008784    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-workload-certs\") pod \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\" (UID: \"c6e1ba27-d171-44ac-b625-2d2e34fb1481\") "
Oct 30 22:51:01.008856 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.008826    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istiod-ca-cert" pod "c6e1ba27-d171-44ac-b625-2d2e34fb1481" (UID: "c6e1ba27-d171-44ac-b625-2d2e34fb1481"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:01.008916 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.008871    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-credential-socket" pod "c6e1ba27-d171-44ac-b625-2d2e34fb1481" (UID: "c6e1ba27-d171-44ac-b625-2d2e34fb1481"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:01.008983 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.008915    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-workload-socket" pod "c6e1ba27-d171-44ac-b625-2d2e34fb1481" (UID: "c6e1ba27-d171-44ac-b625-2d2e34fb1481"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:01.009233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.009209    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-workload-certs" pod "c6e1ba27-d171-44ac-b625-2d2e34fb1481" (UID: "c6e1ba27-d171-44ac-b625-2d2e34fb1481"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:01.009527 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.009388    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:01.009527 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.009407    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:01.009527 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.009421    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:01.009527 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.009435    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:01.009527 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.009448    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:01.010740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.010713    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-token" pod "c6e1ba27-d171-44ac-b625-2d2e34fb1481" (UID: "c6e1ba27-d171-44ac-b625-2d2e34fb1481"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:01.010881 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.010813    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e1ba27-d171-44ac-b625-2d2e34fb1481-kube-api-access-4fbwf" pod "c6e1ba27-d171-44ac-b625-2d2e34fb1481" (UID: "c6e1ba27-d171-44ac-b625-2d2e34fb1481"). InnerVolumeSpecName "kube-api-access-4fbwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:01.010986 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.010968    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-podinfo" pod "c6e1ba27-d171-44ac-b625-2d2e34fb1481" (UID: "c6e1ba27-d171-44ac-b625-2d2e34fb1481"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:51:01.011117 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.011092    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-envoy" pod "c6e1ba27-d171-44ac-b625-2d2e34fb1481" (UID: "c6e1ba27-d171-44ac-b625-2d2e34fb1481"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:01.110600 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.110561    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:01.110600 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.110593    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:01.110829 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.110607    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fbwf\" (UniqueName: \"kubernetes.io/projected/c6e1ba27-d171-44ac-b625-2d2e34fb1481-kube-api-access-4fbwf\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:01.110829 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.110623    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e1ba27-d171-44ac-b625-2d2e34fb1481-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:01.542668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.542632    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"]
Oct 30 22:51:01.672658 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.672560    4553 generic.go:356] "Generic (PLEG): container finished" podID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerID="abb7db58614a51bfd8b6343b285d02e6506c28aff6c399c0b56fb6751db9cb1b" exitCode=0
Oct 30 22:51:01.672658 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.672615    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"
Oct 30 22:51:01.672658 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.672629    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp" event={"ID":"c6e1ba27-d171-44ac-b625-2d2e34fb1481","Type":"ContainerDied","Data":"abb7db58614a51bfd8b6343b285d02e6506c28aff6c399c0b56fb6751db9cb1b"}
Oct 30 22:51:01.672954 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.672673    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp" event={"ID":"c6e1ba27-d171-44ac-b625-2d2e34fb1481","Type":"ContainerDied","Data":"b1236fcc15dcdf264f81d974661e4131fbfb566f63f837e3e45a876a882dadea"}
Oct 30 22:51:01.672954 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.672694    4553 scope.go:122] "RemoveContainer" containerID="abb7db58614a51bfd8b6343b285d02e6506c28aff6c399c0b56fb6751db9cb1b"
Oct 30 22:51:01.683124 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.683105    4553 scope.go:122] "RemoveContainer" containerID="a1f8da1485dd857622bce01acd471fdf203b396d3a42b55d3ab0a9bfefb2056a"
Oct 30 22:51:01.693088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.693067    4553 scope.go:122] "RemoveContainer" containerID="abb7db58614a51bfd8b6343b285d02e6506c28aff6c399c0b56fb6751db9cb1b"
Oct 30 22:51:01.693434 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:51:01.693408    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb7db58614a51bfd8b6343b285d02e6506c28aff6c399c0b56fb6751db9cb1b\": container with ID starting with abb7db58614a51bfd8b6343b285d02e6506c28aff6c399c0b56fb6751db9cb1b not found: ID does not exist" containerID="abb7db58614a51bfd8b6343b285d02e6506c28aff6c399c0b56fb6751db9cb1b"
Oct 30 22:51:01.693524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.693446    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb7db58614a51bfd8b6343b285d02e6506c28aff6c399c0b56fb6751db9cb1b"} err="failed to get container status \"abb7db58614a51bfd8b6343b285d02e6506c28aff6c399c0b56fb6751db9cb1b\": rpc error: code = NotFound desc = could not find container \"abb7db58614a51bfd8b6343b285d02e6506c28aff6c399c0b56fb6751db9cb1b\": container with ID starting with abb7db58614a51bfd8b6343b285d02e6506c28aff6c399c0b56fb6751db9cb1b not found: ID does not exist"
Oct 30 22:51:01.693524 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.693474    4553 scope.go:122] "RemoveContainer" containerID="a1f8da1485dd857622bce01acd471fdf203b396d3a42b55d3ab0a9bfefb2056a"
Oct 30 22:51:01.693726 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:51:01.693707    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f8da1485dd857622bce01acd471fdf203b396d3a42b55d3ab0a9bfefb2056a\": container with ID starting with a1f8da1485dd857622bce01acd471fdf203b396d3a42b55d3ab0a9bfefb2056a not found: ID does not exist" containerID="a1f8da1485dd857622bce01acd471fdf203b396d3a42b55d3ab0a9bfefb2056a"
Oct 30 22:51:01.693778 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.693735    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f8da1485dd857622bce01acd471fdf203b396d3a42b55d3ab0a9bfefb2056a"} err="failed to get container status \"a1f8da1485dd857622bce01acd471fdf203b396d3a42b55d3ab0a9bfefb2056a\": rpc error: code = NotFound desc = could not find container \"a1f8da1485dd857622bce01acd471fdf203b396d3a42b55d3ab0a9bfefb2056a\": container with ID starting with a1f8da1485dd857622bce01acd471fdf203b396d3a42b55d3ab0a9bfefb2056a not found: ID does not exist"
Oct 30 22:51:01.844784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.844745    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"]
Oct 30 22:51:01.850557 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:01.850521    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b4deef-openshift-default-6d68f9f457-v8vrp"]
Oct 30 22:51:03.394375 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:03.394335    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" path="/var/lib/kubelet/pods/c6e1ba27-d171-44ac-b625-2d2e34fb1481/volumes"
Oct 30 22:51:06.543795 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:06.543749    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl" podUID="310c1cdc-3c1b-4d84-9f50-9ac5e59e6198" containerName="payload-processing" containerID="cri-o://bd4b766fb4b7a5e0bfc5edb6084de5a3c8b07e3918fc1996638247d8d53b75e0" gracePeriod=25
Oct 30 22:51:08.161722 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:08.161688    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"]
Oct 30 22:51:13.162597 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:13.162551    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9" podUID="f0886f20-1c73-4f9a-8f1d-63db1a2c5aba" containerName="payload-pre-processing" containerID="cri-o://d11cd22791589be1515f42c4a10201920793d1216597e0246a1aec8a5a6cfd26" gracePeriod=25
Oct 30 22:51:14.714045 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:14.713999    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl" podUID="310c1cdc-3c1b-4d84-9f50-9ac5e59e6198" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:17.376147 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.376104    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"]
Oct 30 22:51:17.376814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.376796    4553 cpu_manager.go:414] "containerMap: removing container" podUID="226981d5-7cad-4f8b-9a87-e3c6bef2e59a" containerName="maas-api"
Oct 30 22:51:17.376860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.376816    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="226981d5-7cad-4f8b-9a87-e3c6bef2e59a" containerName="maas-api"
Oct 30 22:51:17.376860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.376839    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerName="istio-proxy"
Oct 30 22:51:17.376860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.376845    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerName="istio-proxy"
Oct 30 22:51:17.376860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.376852    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerName="istio-proxy"
Oct 30 22:51:17.376860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.376856    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerName="istio-proxy"
Oct 30 22:51:17.377024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.376868    4553 cpu_manager.go:414] "containerMap: removing container" podUID="b2d352d2-8937-4ae7-9c60-2b5324fcea80" containerName="maas-api"
Oct 30 22:51:17.377024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.376873    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d352d2-8937-4ae7-9c60-2b5324fcea80" containerName="maas-api"
Oct 30 22:51:17.377024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.376879    4553 cpu_manager.go:414] "containerMap: removing container" podUID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerName="istio-proxy"
Oct 30 22:51:17.377024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.376884    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerName="istio-proxy"
Oct 30 22:51:17.377024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.376979    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerName="istio-proxy"
Oct 30 22:51:17.377024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.376988    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f83e3e-757f-495d-9734-e3f7a77d9b9d" containerName="istio-proxy"
Oct 30 22:51:17.377024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.376995    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d352d2-8937-4ae7-9c60-2b5324fcea80" containerName="maas-api"
Oct 30 22:51:17.377024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.377002    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerName="istio-proxy"
Oct 30 22:51:17.377024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.377011    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerName="istio-proxy"
Oct 30 22:51:17.377024 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.377017    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="226981d5-7cad-4f8b-9a87-e3c6bef2e59a" containerName="maas-api"
Oct 30 22:51:17.380535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.380519    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:17.417200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.417125    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:17.417393 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.417271    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-metrics-certs\") pod \"maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:17.417447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.417438    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:17.417508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.417457    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf255\" (UniqueName: \"kubernetes.io/projected/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-kube-api-access-cf255\") pod \"maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:17.433730 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.433696    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-6924e5-serving-cert\""
Oct 30 22:51:17.489077 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.489042    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"]
Oct 30 22:51:17.519230 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.519185    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:17.519438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.519243    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf255\" (UniqueName: \"kubernetes.io/projected/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-kube-api-access-cf255\") pod \"maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:17.519438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.519413    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:17.519575 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.519475    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-metrics-certs\") pod \"maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:17.519575 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:51:17.519524    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-serving-cert: secret "maas-api-e2e-extmod-a-6924e5-serving-cert" not found
Oct 30 22:51:17.519682 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:51:17.519632    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-maas-api-tls podName:c4d6c182-7bf5-4a2c-95d5-174b4f34d298 nodeName:}" failed. No retries permitted until 2026-10-30 22:51:18.019601974 +0000 UTC m=+4128.753964997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-maas-api-tls") pod "maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp" (UID: "c4d6c182-7bf5-4a2c-95d5-174b4f34d298") : secret "maas-api-e2e-extmod-a-6924e5-serving-cert" not found
Oct 30 22:51:17.520019 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.519997    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:17.522200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.522171    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-metrics-certs\") pod \"maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:17.578326 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:17.578288    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf255\" (UniqueName: \"kubernetes.io/projected/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-kube-api-access-cf255\") pod \"maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:18.025472 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:18.025428    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:18.028336 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:18.028305    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:18.296572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:18.296475    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:18.632017 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:18.631979    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"]
Oct 30 22:51:18.632529 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:18.632233    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4" podUID="b4a2de90-2ce9-452e-a5ed-63606a25e513" containerName="istio-proxy" containerID="cri-o://6f50dd5e5c5b97a46765121d3b2679038ef8ee212f77288a2eace1bf1de4a904" gracePeriod=30
Oct 30 22:51:18.713803 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:18.713763    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"]
Oct 30 22:51:18.723377 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:18.722320    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:51:18.790110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:18.790074    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp" event={"ID":"c4d6c182-7bf5-4a2c-95d5-174b4f34d298","Type":"ContainerStarted","Data":"51d2a9e93841d4bd7748b8a79f7e0384e5d6c0807662f7b0c5c80d5efa8dafb6"}
Oct 30 22:51:19.798684 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:19.798637    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp" event={"ID":"c4d6c182-7bf5-4a2c-95d5-174b4f34d298","Type":"ContainerStarted","Data":"c7458f2586909bbced7745efe43f8219abe4739f3056f5d330faa1fa5c0eafac"}
Oct 30 22:51:19.799278 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:19.798717    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:19.946113 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:19.946003    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp" podStartSLOduration=2.031219845 podStartE2EDuration="2.945986645s" totalImagesPullingTime="914.7668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:19.943486663 +0000 UTC m=+4130.677849706" watchObservedRunningTime="2026-10-30 22:51:19.945986645 +0000 UTC m=+4130.680349734"
Oct 30 22:51:20.426400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:20.426352    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9" podUID="f0886f20-1c73-4f9a-8f1d-63db1a2c5aba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:22.222361 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:22.222318    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"]
Oct 30 22:51:22.222795 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:22.222531    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb" podUID="d8da6201-f247-4182-a143-6a880f98b175" containerName="istio-proxy" containerID="cri-o://eb791c5ae4268f4a917b3efe2db8399aba20ae4063af6c1fe930f154f78ea033" gracePeriod=30
Oct 30 22:51:23.829578 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.829541    4553 generic.go:356] "Generic (PLEG): container finished" podID="b4a2de90-2ce9-452e-a5ed-63606a25e513" containerID="6f50dd5e5c5b97a46765121d3b2679038ef8ee212f77288a2eace1bf1de4a904" exitCode=0
Oct 30 22:51:23.830073 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.829695    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4" event={"ID":"b4a2de90-2ce9-452e-a5ed-63606a25e513","Type":"ContainerDied","Data":"6f50dd5e5c5b97a46765121d3b2679038ef8ee212f77288a2eace1bf1de4a904"}
Oct 30 22:51:23.830073 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.829753    4553 scope.go:122] "RemoveContainer" containerID="038c8522d1a99a4d19c4c82931bfca77d880ae97b3668c195fa5571d6b0b04a6"
Oct 30 22:51:23.904995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.904921    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:51:23.999106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.999067    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-workload-socket\") pod \"b4a2de90-2ce9-452e-a5ed-63606a25e513\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") "
Oct 30 22:51:23.999353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.999121    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-workload-certs\") pod \"b4a2de90-2ce9-452e-a5ed-63606a25e513\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") "
Oct 30 22:51:23.999353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.999185    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a2de90-2ce9-452e-a5ed-63606a25e513-kube-api-access-kj4bg\" (UniqueName: \"kubernetes.io/projected/b4a2de90-2ce9-452e-a5ed-63606a25e513-kube-api-access-kj4bg\") pod \"b4a2de90-2ce9-452e-a5ed-63606a25e513\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") "
Oct 30 22:51:23.999353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.999226    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-credential-socket\") pod \"b4a2de90-2ce9-452e-a5ed-63606a25e513\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") "
Oct 30 22:51:23.999353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.999303    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a2de90-2ce9-452e-a5ed-63606a25e513-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4a2de90-2ce9-452e-a5ed-63606a25e513-istiod-ca-cert\") pod \"b4a2de90-2ce9-452e-a5ed-63606a25e513\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") "
Oct 30 22:51:23.999353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.999322    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-token\" (UniqueName: \"kubernetes.io/projected/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-token\") pod \"b4a2de90-2ce9-452e-a5ed-63606a25e513\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") "
Oct 30 22:51:23.999653 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.999384    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-envoy\") pod \"b4a2de90-2ce9-452e-a5ed-63606a25e513\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") "
Oct 30 22:51:23.999653 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.999435    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-workload-socket" pod "b4a2de90-2ce9-452e-a5ed-63606a25e513" (UID: "b4a2de90-2ce9-452e-a5ed-63606a25e513"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:23.999653 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.999474    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-data\") pod \"b4a2de90-2ce9-452e-a5ed-63606a25e513\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") "
Oct 30 22:51:23.999653 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.999492    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-workload-certs" pod "b4a2de90-2ce9-452e-a5ed-63606a25e513" (UID: "b4a2de90-2ce9-452e-a5ed-63606a25e513"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:23.999653 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.999620    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-credential-socket" pod "b4a2de90-2ce9-452e-a5ed-63606a25e513" (UID: "b4a2de90-2ce9-452e-a5ed-63606a25e513"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:23.999937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.999786    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-data" pod "b4a2de90-2ce9-452e-a5ed-63606a25e513" (UID: "b4a2de90-2ce9-452e-a5ed-63606a25e513"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:23.999937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.999589    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-podinfo\") pod \"b4a2de90-2ce9-452e-a5ed-63606a25e513\" (UID: \"b4a2de90-2ce9-452e-a5ed-63606a25e513\") "
Oct 30 22:51:23.999937 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:23.999815    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a2de90-2ce9-452e-a5ed-63606a25e513-istiod-ca-cert" pod "b4a2de90-2ce9-452e-a5ed-63606a25e513" (UID: "b4a2de90-2ce9-452e-a5ed-63606a25e513"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:24.000597 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.000573    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:24.000697 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.000601    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:24.000697 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.000617    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:24.000697 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.000632    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4a2de90-2ce9-452e-a5ed-63606a25e513-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:24.000697 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.000646    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:24.001872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.001842    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-token" pod "b4a2de90-2ce9-452e-a5ed-63606a25e513" (UID: "b4a2de90-2ce9-452e-a5ed-63606a25e513"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:24.001983 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.001872    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-podinfo" pod "b4a2de90-2ce9-452e-a5ed-63606a25e513" (UID: "b4a2de90-2ce9-452e-a5ed-63606a25e513"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:51:24.002125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.002105    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-envoy" pod "b4a2de90-2ce9-452e-a5ed-63606a25e513" (UID: "b4a2de90-2ce9-452e-a5ed-63606a25e513"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:24.002256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.002186    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a2de90-2ce9-452e-a5ed-63606a25e513-kube-api-access-kj4bg" pod "b4a2de90-2ce9-452e-a5ed-63606a25e513" (UID: "b4a2de90-2ce9-452e-a5ed-63606a25e513"). InnerVolumeSpecName "kube-api-access-kj4bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:24.102108 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.102070    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:24.102108 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.102106    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj4bg\" (UniqueName: \"kubernetes.io/projected/b4a2de90-2ce9-452e-a5ed-63606a25e513-kube-api-access-kj4bg\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:24.102108 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.102117    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:24.102463 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.102126    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4a2de90-2ce9-452e-a5ed-63606a25e513-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:24.714373 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.714333    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl" podUID="310c1cdc-3c1b-4d84-9f50-9ac5e59e6198" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:24.840686 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.840644    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"
Oct 30 22:51:24.841213 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.840638    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4" event={"ID":"b4a2de90-2ce9-452e-a5ed-63606a25e513","Type":"ContainerDied","Data":"02a1205f02a60959f65025e01534e68bb6fb1ed6f1a4c9eec781dc17b3e967e1"}
Oct 30 22:51:24.841213 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:24.840793    4553 scope.go:122] "RemoveContainer" containerID="6f50dd5e5c5b97a46765121d3b2679038ef8ee212f77288a2eace1bf1de4a904"
Oct 30 22:51:25.012074 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:25.011987    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"]
Oct 30 22:51:25.062417 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:25.062370    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b1b1eb-openshift-default-75fb95c585-cf7s4"]
Oct 30 22:51:25.406949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:25.406526    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a2de90-2ce9-452e-a5ed-63606a25e513" path="/var/lib/kubelet/pods/b4a2de90-2ce9-452e-a5ed-63606a25e513/volumes"
Oct 30 22:51:25.811059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:25.810980    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:51:26.830507 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:26.830482    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"
Oct 30 22:51:26.857946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:26.857913    4553 generic.go:356] "Generic (PLEG): container finished" podID="310c1cdc-3c1b-4d84-9f50-9ac5e59e6198" containerID="bd4b766fb4b7a5e0bfc5edb6084de5a3c8b07e3918fc1996638247d8d53b75e0" exitCode=0
Oct 30 22:51:26.858190 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:26.857960    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"
Oct 30 22:51:26.858190 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:26.857972    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl" event={"ID":"310c1cdc-3c1b-4d84-9f50-9ac5e59e6198","Type":"ContainerDied","Data":"bd4b766fb4b7a5e0bfc5edb6084de5a3c8b07e3918fc1996638247d8d53b75e0"}
Oct 30 22:51:26.858190 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:26.858012    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl" event={"ID":"310c1cdc-3c1b-4d84-9f50-9ac5e59e6198","Type":"ContainerDied","Data":"37f3bfbe84e72ccf88bbea798f9825c13a403044a2b28a871260df3778ba7fd1"}
Oct 30 22:51:26.858190 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:26.858038    4553 scope.go:122] "RemoveContainer" containerID="bd4b766fb4b7a5e0bfc5edb6084de5a3c8b07e3918fc1996638247d8d53b75e0"
Oct 30 22:51:26.867892 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:26.867872    4553 scope.go:122] "RemoveContainer" containerID="bd4b766fb4b7a5e0bfc5edb6084de5a3c8b07e3918fc1996638247d8d53b75e0"
Oct 30 22:51:26.868190 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:51:26.868169    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4b766fb4b7a5e0bfc5edb6084de5a3c8b07e3918fc1996638247d8d53b75e0\": container with ID starting with bd4b766fb4b7a5e0bfc5edb6084de5a3c8b07e3918fc1996638247d8d53b75e0 not found: ID does not exist" containerID="bd4b766fb4b7a5e0bfc5edb6084de5a3c8b07e3918fc1996638247d8d53b75e0"
Oct 30 22:51:26.868269 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:26.868197    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4b766fb4b7a5e0bfc5edb6084de5a3c8b07e3918fc1996638247d8d53b75e0"} err="failed to get container status \"bd4b766fb4b7a5e0bfc5edb6084de5a3c8b07e3918fc1996638247d8d53b75e0\": rpc error: code = NotFound desc = could not find container \"bd4b766fb4b7a5e0bfc5edb6084de5a3c8b07e3918fc1996638247d8d53b75e0\": container with ID starting with bd4b766fb4b7a5e0bfc5edb6084de5a3c8b07e3918fc1996638247d8d53b75e0 not found: ID does not exist"
Oct 30 22:51:26.938237 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:26.938199    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310c1cdc-3c1b-4d84-9f50-9ac5e59e6198-kube-api-access-cdhch\" (UniqueName: \"kubernetes.io/projected/310c1cdc-3c1b-4d84-9f50-9ac5e59e6198-kube-api-access-cdhch\") pod \"310c1cdc-3c1b-4d84-9f50-9ac5e59e6198\" (UID: \"310c1cdc-3c1b-4d84-9f50-9ac5e59e6198\") "
Oct 30 22:51:26.938462 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:26.938305    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/310c1cdc-3c1b-4d84-9f50-9ac5e59e6198-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310c1cdc-3c1b-4d84-9f50-9ac5e59e6198-plugins-config-volume\") pod \"310c1cdc-3c1b-4d84-9f50-9ac5e59e6198\" (UID: \"310c1cdc-3c1b-4d84-9f50-9ac5e59e6198\") "
Oct 30 22:51:26.938751 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:26.938717    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310c1cdc-3c1b-4d84-9f50-9ac5e59e6198-plugins-config-volume" pod "310c1cdc-3c1b-4d84-9f50-9ac5e59e6198" (UID: "310c1cdc-3c1b-4d84-9f50-9ac5e59e6198"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:26.940614 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:26.940592    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310c1cdc-3c1b-4d84-9f50-9ac5e59e6198-kube-api-access-cdhch" pod "310c1cdc-3c1b-4d84-9f50-9ac5e59e6198" (UID: "310c1cdc-3c1b-4d84-9f50-9ac5e59e6198"). InnerVolumeSpecName "kube-api-access-cdhch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:27.040556 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.040512    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310c1cdc-3c1b-4d84-9f50-9ac5e59e6198-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:27.040556 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.040546    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdhch\" (UniqueName: \"kubernetes.io/projected/310c1cdc-3c1b-4d84-9f50-9ac5e59e6198-kube-api-access-cdhch\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:27.281491 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.281455    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"]
Oct 30 22:51:27.284007 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.283985    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a0116e-5475dd76fc-hs2vl"]
Oct 30 22:51:27.394591 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.394555    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="310c1cdc-3c1b-4d84-9f50-9ac5e59e6198" path="/var/lib/kubelet/pods/310c1cdc-3c1b-4d84-9f50-9ac5e59e6198/volumes"
Oct 30 22:51:27.488445 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.488418    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:51:27.540149 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.540108    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"]
Oct 30 22:51:27.545044 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.545015    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-workload-certs\") pod \"d8da6201-f247-4182-a143-6a880f98b175\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") "
Oct 30 22:51:27.545276 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.545083    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-workload-socket\") pod \"d8da6201-f247-4182-a143-6a880f98b175\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") "
Oct 30 22:51:27.545276 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.545178    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-istio-envoy\") pod \"d8da6201-f247-4182-a143-6a880f98b175\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") "
Oct 30 22:51:27.545276 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.545201    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8da6201-f247-4182-a143-6a880f98b175-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8da6201-f247-4182-a143-6a880f98b175-istiod-ca-cert\") pod \"d8da6201-f247-4182-a143-6a880f98b175\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") "
Oct 30 22:51:27.545276 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.545240    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8da6201-f247-4182-a143-6a880f98b175-istio-token\" (UniqueName: \"kubernetes.io/projected/d8da6201-f247-4182-a143-6a880f98b175-istio-token\") pod \"d8da6201-f247-4182-a143-6a880f98b175\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") "
Oct 30 22:51:27.545530 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.545297    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-credential-socket\") pod \"d8da6201-f247-4182-a143-6a880f98b175\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") "
Oct 30 22:51:27.545530 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.545329    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8da6201-f247-4182-a143-6a880f98b175-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8da6201-f247-4182-a143-6a880f98b175-istio-podinfo\") pod \"d8da6201-f247-4182-a143-6a880f98b175\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") "
Oct 30 22:51:27.545530 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.545335    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-workload-certs" pod "d8da6201-f247-4182-a143-6a880f98b175" (UID: "d8da6201-f247-4182-a143-6a880f98b175"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:27.545530 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.545415    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8da6201-f247-4182-a143-6a880f98b175-kube-api-access-rzj55\" (UniqueName: \"kubernetes.io/projected/d8da6201-f247-4182-a143-6a880f98b175-kube-api-access-rzj55\") pod \"d8da6201-f247-4182-a143-6a880f98b175\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") "
Oct 30 22:51:27.545530 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.545438    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-workload-socket" pod "d8da6201-f247-4182-a143-6a880f98b175" (UID: "d8da6201-f247-4182-a143-6a880f98b175"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:27.545530 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.545460    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-istio-data\") pod \"d8da6201-f247-4182-a143-6a880f98b175\" (UID: \"d8da6201-f247-4182-a143-6a880f98b175\") "
Oct 30 22:51:27.545958 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.545668    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8da6201-f247-4182-a143-6a880f98b175-istiod-ca-cert" pod "d8da6201-f247-4182-a143-6a880f98b175" (UID: "d8da6201-f247-4182-a143-6a880f98b175"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:27.545958 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.545748    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-istio-data" pod "d8da6201-f247-4182-a143-6a880f98b175" (UID: "d8da6201-f247-4182-a143-6a880f98b175"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:27.546178 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.546151    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-credential-socket" pod "d8da6201-f247-4182-a143-6a880f98b175" (UID: "d8da6201-f247-4182-a143-6a880f98b175"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:27.546178 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.546157    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8da6201-f247-4182-a143-6a880f98b175-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:27.546341 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.546187    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:27.546341 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.546199    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:27.546341 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.546208    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:27.547882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.547858    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8da6201-f247-4182-a143-6a880f98b175-kube-api-access-rzj55" pod "d8da6201-f247-4182-a143-6a880f98b175" (UID: "d8da6201-f247-4182-a143-6a880f98b175"). InnerVolumeSpecName "kube-api-access-rzj55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:27.547998 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.547978    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8da6201-f247-4182-a143-6a880f98b175-istio-token" pod "d8da6201-f247-4182-a143-6a880f98b175" (UID: "d8da6201-f247-4182-a143-6a880f98b175"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:27.548634 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.548614    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-istio-envoy" pod "d8da6201-f247-4182-a143-6a880f98b175" (UID: "d8da6201-f247-4182-a143-6a880f98b175"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:27.548634 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.548621    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8da6201-f247-4182-a143-6a880f98b175-istio-podinfo" pod "d8da6201-f247-4182-a143-6a880f98b175" (UID: "d8da6201-f247-4182-a143-6a880f98b175"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:51:27.589788 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.589748    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"]
Oct 30 22:51:27.593072 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.593041    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"]
Oct 30 22:51:27.647015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.646981    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:27.647015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.647010    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8da6201-f247-4182-a143-6a880f98b175-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:27.647015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.647023    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da6201-f247-4182-a143-6a880f98b175-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:27.647425 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.647037    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8da6201-f247-4182-a143-6a880f98b175-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:27.647425 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.647048    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzj55\" (UniqueName: \"kubernetes.io/projected/d8da6201-f247-4182-a143-6a880f98b175-kube-api-access-rzj55\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:27.866614 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.866578    4553 generic.go:356] "Generic (PLEG): container finished" podID="d8da6201-f247-4182-a143-6a880f98b175" containerID="eb791c5ae4268f4a917b3efe2db8399aba20ae4063af6c1fe930f154f78ea033" exitCode=0
Oct 30 22:51:27.867123 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.866628    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"
Oct 30 22:51:27.867123 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.866662    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb" event={"ID":"d8da6201-f247-4182-a143-6a880f98b175","Type":"ContainerDied","Data":"eb791c5ae4268f4a917b3efe2db8399aba20ae4063af6c1fe930f154f78ea033"}
Oct 30 22:51:27.867123 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.866727    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb" event={"ID":"d8da6201-f247-4182-a143-6a880f98b175","Type":"ContainerDied","Data":"d867fd48754bd87a8de7686a072d0e5018542a9b7d53b603b7676f32f54209be"}
Oct 30 22:51:27.867123 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.866754    4553 scope.go:122] "RemoveContainer" containerID="eb791c5ae4268f4a917b3efe2db8399aba20ae4063af6c1fe930f154f78ea033"
Oct 30 22:51:27.877649 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.877631    4553 scope.go:122] "RemoveContainer" containerID="60e8df7b826a53be9952143308a4038986302ccdc984113e905ac6720b17a472"
Oct 30 22:51:27.887754 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.887735    4553 scope.go:122] "RemoveContainer" containerID="eb791c5ae4268f4a917b3efe2db8399aba20ae4063af6c1fe930f154f78ea033"
Oct 30 22:51:27.888036 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:51:27.888017    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb791c5ae4268f4a917b3efe2db8399aba20ae4063af6c1fe930f154f78ea033\": container with ID starting with eb791c5ae4268f4a917b3efe2db8399aba20ae4063af6c1fe930f154f78ea033 not found: ID does not exist" containerID="eb791c5ae4268f4a917b3efe2db8399aba20ae4063af6c1fe930f154f78ea033"
Oct 30 22:51:27.888081 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.888049    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb791c5ae4268f4a917b3efe2db8399aba20ae4063af6c1fe930f154f78ea033"} err="failed to get container status \"eb791c5ae4268f4a917b3efe2db8399aba20ae4063af6c1fe930f154f78ea033\": rpc error: code = NotFound desc = could not find container \"eb791c5ae4268f4a917b3efe2db8399aba20ae4063af6c1fe930f154f78ea033\": container with ID starting with eb791c5ae4268f4a917b3efe2db8399aba20ae4063af6c1fe930f154f78ea033 not found: ID does not exist"
Oct 30 22:51:27.888081 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.888068    4553 scope.go:122] "RemoveContainer" containerID="60e8df7b826a53be9952143308a4038986302ccdc984113e905ac6720b17a472"
Oct 30 22:51:27.888337 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:51:27.888320    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e8df7b826a53be9952143308a4038986302ccdc984113e905ac6720b17a472\": container with ID starting with 60e8df7b826a53be9952143308a4038986302ccdc984113e905ac6720b17a472 not found: ID does not exist" containerID="60e8df7b826a53be9952143308a4038986302ccdc984113e905ac6720b17a472"
Oct 30 22:51:27.888388 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:27.888341    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e8df7b826a53be9952143308a4038986302ccdc984113e905ac6720b17a472"} err="failed to get container status \"60e8df7b826a53be9952143308a4038986302ccdc984113e905ac6720b17a472\": rpc error: code = NotFound desc = could not find container \"60e8df7b826a53be9952143308a4038986302ccdc984113e905ac6720b17a472\": container with ID starting with 60e8df7b826a53be9952143308a4038986302ccdc984113e905ac6720b17a472 not found: ID does not exist"
Oct 30 22:51:28.076864 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:28.076829    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"]
Oct 30 22:51:28.081124 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:28.081094    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-9f7654-openshift-default-697764d49-hmjxb"]
Oct 30 22:51:28.202986 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:51:28.202893    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:51:29.394227 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:29.394190    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8da6201-f247-4182-a143-6a880f98b175" path="/var/lib/kubelet/pods/d8da6201-f247-4182-a143-6a880f98b175/volumes"
Oct 30 22:51:30.427110 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:30.427066    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9" podUID="f0886f20-1c73-4f9a-8f1d-63db1a2c5aba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:31.042862 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:31.042821    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"]
Oct 30 22:51:31.147408 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:31.147353    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"]
Oct 30 22:51:33.435093 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:33.435025    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"
Oct 30 22:51:33.521622 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:33.521585    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0886f20-1c73-4f9a-8f1d-63db1a2c5aba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0886f20-1c73-4f9a-8f1d-63db1a2c5aba-plugins-config-volume\") pod \"f0886f20-1c73-4f9a-8f1d-63db1a2c5aba\" (UID: \"f0886f20-1c73-4f9a-8f1d-63db1a2c5aba\") "
Oct 30 22:51:33.521844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:33.521773    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0886f20-1c73-4f9a-8f1d-63db1a2c5aba-kube-api-access-68wbp\" (UniqueName: \"kubernetes.io/projected/f0886f20-1c73-4f9a-8f1d-63db1a2c5aba-kube-api-access-68wbp\") pod \"f0886f20-1c73-4f9a-8f1d-63db1a2c5aba\" (UID: \"f0886f20-1c73-4f9a-8f1d-63db1a2c5aba\") "
Oct 30 22:51:33.522008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:33.521982    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0886f20-1c73-4f9a-8f1d-63db1a2c5aba-plugins-config-volume" pod "f0886f20-1c73-4f9a-8f1d-63db1a2c5aba" (UID: "f0886f20-1c73-4f9a-8f1d-63db1a2c5aba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:33.522256 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:33.522238    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0886f20-1c73-4f9a-8f1d-63db1a2c5aba-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:33.524233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:33.524207    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0886f20-1c73-4f9a-8f1d-63db1a2c5aba-kube-api-access-68wbp" pod "f0886f20-1c73-4f9a-8f1d-63db1a2c5aba" (UID: "f0886f20-1c73-4f9a-8f1d-63db1a2c5aba"). InnerVolumeSpecName "kube-api-access-68wbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:33.623565 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:33.623534    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68wbp\" (UniqueName: \"kubernetes.io/projected/f0886f20-1c73-4f9a-8f1d-63db1a2c5aba-kube-api-access-68wbp\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:33.908865 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:33.908828    4553 generic.go:356] "Generic (PLEG): container finished" podID="f0886f20-1c73-4f9a-8f1d-63db1a2c5aba" containerID="d11cd22791589be1515f42c4a10201920793d1216597e0246a1aec8a5a6cfd26" exitCode=0
Oct 30 22:51:33.909062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:33.908896    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"
Oct 30 22:51:33.909062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:33.908891    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9" event={"ID":"f0886f20-1c73-4f9a-8f1d-63db1a2c5aba","Type":"ContainerDied","Data":"d11cd22791589be1515f42c4a10201920793d1216597e0246a1aec8a5a6cfd26"}
Oct 30 22:51:33.909062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:33.909056    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9" event={"ID":"f0886f20-1c73-4f9a-8f1d-63db1a2c5aba","Type":"ContainerDied","Data":"0f29096fa4249110f4726f200a5b9a33d7d53483e886f682a33181597b95b45b"}
Oct 30 22:51:33.909250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:33.909087    4553 scope.go:122] "RemoveContainer" containerID="d11cd22791589be1515f42c4a10201920793d1216597e0246a1aec8a5a6cfd26"
Oct 30 22:51:33.920206 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:33.920126    4553 scope.go:122] "RemoveContainer" containerID="d11cd22791589be1515f42c4a10201920793d1216597e0246a1aec8a5a6cfd26"
Oct 30 22:51:33.920491 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:51:33.920470    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d11cd22791589be1515f42c4a10201920793d1216597e0246a1aec8a5a6cfd26\": container with ID starting with d11cd22791589be1515f42c4a10201920793d1216597e0246a1aec8a5a6cfd26 not found: ID does not exist" containerID="d11cd22791589be1515f42c4a10201920793d1216597e0246a1aec8a5a6cfd26"
Oct 30 22:51:33.920559 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:33.920505    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d11cd22791589be1515f42c4a10201920793d1216597e0246a1aec8a5a6cfd26"} err="failed to get container status \"d11cd22791589be1515f42c4a10201920793d1216597e0246a1aec8a5a6cfd26\": rpc error: code = NotFound desc = could not find container \"d11cd22791589be1515f42c4a10201920793d1216597e0246a1aec8a5a6cfd26\": container with ID starting with d11cd22791589be1515f42c4a10201920793d1216597e0246a1aec8a5a6cfd26 not found: ID does not exist"
Oct 30 22:51:34.078577 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:34.078543    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"]
Oct 30 22:51:34.129216 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:34.129175    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f7654-5b8fc8fc7f-b2dz9"]
Oct 30 22:51:35.393516 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:35.393484    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0886f20-1c73-4f9a-8f1d-63db1a2c5aba" path="/var/lib/kubelet/pods/f0886f20-1c73-4f9a-8f1d-63db1a2c5aba/volumes"
Oct 30 22:51:36.043703 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:36.043649    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t" podUID="417967ed-ef2e-449f-95fe-fadf4e6926e5" containerName="payload-processing" containerID="cri-o://fc50e97c7b83bf97fe00fee61b59d0ca938a37ea0655cd5823fa6a8584db2500" gracePeriod=25
Oct 30 22:51:36.148675 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:36.148632    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q" podUID="8a3d82a4-6522-40f9-a000-e92b7b95aa76" containerName="payload-pre-processing" containerID="cri-o://de16a9c2f1639636b15130b2e6af76bf9b6ef4edb98b82e0efbcc5ce9c2de3d0" gracePeriod=25
Oct 30 22:51:42.568603 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.568540    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" containerID="cri-o://a1cb13aab1c59d7a551a4e5a46b14c85c0be3f60b1397715a50b29b7be145199" gracePeriod=45
Oct 30 22:51:42.611725 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.611686    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" containerID="cri-o://9dc2dc024565c85f118a71439638e356e28a603bc750fbb0716f7375b88b29cb" gracePeriod=45
Oct 30 22:51:42.614668 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.614634    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main" containerID="cri-o://62fc6fe60fc088cb27ee0b57043ac310bd52cdc63860dd1d144c076c9c505d53" gracePeriod=45
Oct 30 22:51:42.799352 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.797746    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": dial tcp 10.134.0.68:8000: connect: connection refused"
Oct 30 22:51:42.948020 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.947986    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:51:42.952154 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.952115    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:51:42.956605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.956579    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 22:51:42.972198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.972164    4553 generic.go:356] "Generic (PLEG): container finished" podID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerID="62fc6fe60fc088cb27ee0b57043ac310bd52cdc63860dd1d144c076c9c505d53" exitCode=0
Oct 30 22:51:42.972375 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.972249    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"
Oct 30 22:51:42.972375 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.972277    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" event={"ID":"86ecfcbe-143b-46ef-ac5f-fed921354ef6","Type":"ContainerDied","Data":"62fc6fe60fc088cb27ee0b57043ac310bd52cdc63860dd1d144c076c9c505d53"}
Oct 30 22:51:42.972375 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.972322    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89" event={"ID":"86ecfcbe-143b-46ef-ac5f-fed921354ef6","Type":"ContainerDied","Data":"24df1a8a7c0d1fc09e09058908577ef04e297b15878152610ea281ce23ef4ebf"}
Oct 30 22:51:42.972375 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.972344    4553 scope.go:122] "RemoveContainer" containerID="62fc6fe60fc088cb27ee0b57043ac310bd52cdc63860dd1d144c076c9c505d53"
Oct 30 22:51:42.973915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.973897    4553 generic.go:356] "Generic (PLEG): container finished" podID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerID="9dc2dc024565c85f118a71439638e356e28a603bc750fbb0716f7375b88b29cb" exitCode=0
Oct 30 22:51:42.974124 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.974080    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"
Oct 30 22:51:42.974308 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.974230    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" event={"ID":"d3f74f0b-774f-4b9f-ada4-165542267ef4","Type":"ContainerDied","Data":"9dc2dc024565c85f118a71439638e356e28a603bc750fbb0716f7375b88b29cb"}
Oct 30 22:51:42.974308 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.974258    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9" event={"ID":"d3f74f0b-774f-4b9f-ada4-165542267ef4","Type":"ContainerDied","Data":"d11cce11eb61992a081dcd841c5e13567ac4f524a661dc3350de87b76360d020"}
Oct 30 22:51:42.975910 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.975883    4553 generic.go:356] "Generic (PLEG): container finished" podID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerID="a1cb13aab1c59d7a551a4e5a46b14c85c0be3f60b1397715a50b29b7be145199" exitCode=0
Oct 30 22:51:42.976041 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.975928    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" event={"ID":"4925f432-7a20-48b1-a052-d8ca2fd156e8","Type":"ContainerDied","Data":"a1cb13aab1c59d7a551a4e5a46b14c85c0be3f60b1397715a50b29b7be145199"}
Oct 30 22:51:42.976041 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.975955    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" event={"ID":"4925f432-7a20-48b1-a052-d8ca2fd156e8","Type":"ContainerDied","Data":"01c71dd029c4ca925488f2264dc47e5e9392934a8b678eb77ab953795df1e26c"}
Oct 30 22:51:42.976041 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.975965    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"
Oct 30 22:51:42.986320 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.986296    4553 scope.go:122] "RemoveContainer" containerID="62fc6fe60fc088cb27ee0b57043ac310bd52cdc63860dd1d144c076c9c505d53"
Oct 30 22:51:42.986936 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:51:42.986915    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62fc6fe60fc088cb27ee0b57043ac310bd52cdc63860dd1d144c076c9c505d53\": container with ID starting with 62fc6fe60fc088cb27ee0b57043ac310bd52cdc63860dd1d144c076c9c505d53 not found: ID does not exist" containerID="62fc6fe60fc088cb27ee0b57043ac310bd52cdc63860dd1d144c076c9c505d53"
Oct 30 22:51:42.987041 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.986944    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62fc6fe60fc088cb27ee0b57043ac310bd52cdc63860dd1d144c076c9c505d53"} err="failed to get container status \"62fc6fe60fc088cb27ee0b57043ac310bd52cdc63860dd1d144c076c9c505d53\": rpc error: code = NotFound desc = could not find container \"62fc6fe60fc088cb27ee0b57043ac310bd52cdc63860dd1d144c076c9c505d53\": container with ID starting with 62fc6fe60fc088cb27ee0b57043ac310bd52cdc63860dd1d144c076c9c505d53 not found: ID does not exist"
Oct 30 22:51:42.987041 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.986961    4553 scope.go:122] "RemoveContainer" containerID="9dc2dc024565c85f118a71439638e356e28a603bc750fbb0716f7375b88b29cb"
Oct 30 22:51:42.997090 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.997067    4553 scope.go:122] "RemoveContainer" containerID="9dc2dc024565c85f118a71439638e356e28a603bc750fbb0716f7375b88b29cb"
Oct 30 22:51:42.997448 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:51:42.997426    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc2dc024565c85f118a71439638e356e28a603bc750fbb0716f7375b88b29cb\": container with ID starting with 9dc2dc024565c85f118a71439638e356e28a603bc750fbb0716f7375b88b29cb not found: ID does not exist" containerID="9dc2dc024565c85f118a71439638e356e28a603bc750fbb0716f7375b88b29cb"
Oct 30 22:51:42.997526 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.997458    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc2dc024565c85f118a71439638e356e28a603bc750fbb0716f7375b88b29cb"} err="failed to get container status \"9dc2dc024565c85f118a71439638e356e28a603bc750fbb0716f7375b88b29cb\": rpc error: code = NotFound desc = could not find container \"9dc2dc024565c85f118a71439638e356e28a603bc750fbb0716f7375b88b29cb\": container with ID starting with 9dc2dc024565c85f118a71439638e356e28a603bc750fbb0716f7375b88b29cb not found: ID does not exist"
Oct 30 22:51:42.997526 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:42.997477    4553 scope.go:122] "RemoveContainer" containerID="a1cb13aab1c59d7a551a4e5a46b14c85c0be3f60b1397715a50b29b7be145199"
Oct 30 22:51:43.007051 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.007021    4553 scope.go:122] "RemoveContainer" containerID="a1cb13aab1c59d7a551a4e5a46b14c85c0be3f60b1397715a50b29b7be145199"
Oct 30 22:51:43.007383 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:51:43.007356    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1cb13aab1c59d7a551a4e5a46b14c85c0be3f60b1397715a50b29b7be145199\": container with ID starting with a1cb13aab1c59d7a551a4e5a46b14c85c0be3f60b1397715a50b29b7be145199 not found: ID does not exist" containerID="a1cb13aab1c59d7a551a4e5a46b14c85c0be3f60b1397715a50b29b7be145199"
Oct 30 22:51:43.007455 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.007396    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1cb13aab1c59d7a551a4e5a46b14c85c0be3f60b1397715a50b29b7be145199"} err="failed to get container status \"a1cb13aab1c59d7a551a4e5a46b14c85c0be3f60b1397715a50b29b7be145199\": rpc error: code = NotFound desc = could not find container \"a1cb13aab1c59d7a551a4e5a46b14c85c0be3f60b1397715a50b29b7be145199\": container with ID starting with a1cb13aab1c59d7a551a4e5a46b14c85c0be3f60b1397715a50b29b7be145199 not found: ID does not exist"
Oct 30 22:51:43.037813 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.037783    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-tmp-dir\") pod \"4925f432-7a20-48b1-a052-d8ca2fd156e8\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") "
Oct 30 22:51:43.037991 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.037821    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f74f0b-774f-4b9f-ada4-165542267ef4-kube-api-access-5vjwk\" (UniqueName: \"kubernetes.io/projected/d3f74f0b-774f-4b9f-ada4-165542267ef4-kube-api-access-5vjwk\") pod \"d3f74f0b-774f-4b9f-ada4-165542267ef4\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") "
Oct 30 22:51:43.037991 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.037852    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-tmp-dir\") pod \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") "
Oct 30 22:51:43.037991 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.037887    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-home\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-home\") pod \"4925f432-7a20-48b1-a052-d8ca2fd156e8\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") "
Oct 30 22:51:43.037991 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.037943    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-home\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-home\") pod \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") "
Oct 30 22:51:43.038229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038002    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-model-cache\") pod \"d3f74f0b-774f-4b9f-ada4-165542267ef4\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") "
Oct 30 22:51:43.038229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038031    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ecfcbe-143b-46ef-ac5f-fed921354ef6-kube-api-access-phsqq\" (UniqueName: \"kubernetes.io/projected/86ecfcbe-143b-46ef-ac5f-fed921354ef6-kube-api-access-phsqq\") pod \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") "
Oct 30 22:51:43.038229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038058    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-model-cache\") pod \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") "
Oct 30 22:51:43.038229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038094    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4925f432-7a20-48b1-a052-d8ca2fd156e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/4925f432-7a20-48b1-a052-d8ca2fd156e8-tls-certs\") pod \"4925f432-7a20-48b1-a052-d8ca2fd156e8\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") "
Oct 30 22:51:43.038229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038113    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-tmp-dir" pod "4925f432-7a20-48b1-a052-d8ca2fd156e8" (UID: "4925f432-7a20-48b1-a052-d8ca2fd156e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:43.038229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038181    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-home" pod "4925f432-7a20-48b1-a052-d8ca2fd156e8" (UID: "4925f432-7a20-48b1-a052-d8ca2fd156e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:43.038229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038194    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-tmp-dir" pod "86ecfcbe-143b-46ef-ac5f-fed921354ef6" (UID: "86ecfcbe-143b-46ef-ac5f-fed921354ef6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:43.038229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038210    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3f74f0b-774f-4b9f-ada4-165542267ef4-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f74f0b-774f-4b9f-ada4-165542267ef4-tls-certs\") pod \"d3f74f0b-774f-4b9f-ada4-165542267ef4\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") "
Oct 30 22:51:43.038229 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038227    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-home" pod "86ecfcbe-143b-46ef-ac5f-fed921354ef6" (UID: "86ecfcbe-143b-46ef-ac5f-fed921354ef6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:43.038750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038255    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4925f432-7a20-48b1-a052-d8ca2fd156e8-kube-api-access-w8ldh\" (UniqueName: \"kubernetes.io/projected/4925f432-7a20-48b1-a052-d8ca2fd156e8-kube-api-access-w8ldh\") pod \"4925f432-7a20-48b1-a052-d8ca2fd156e8\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") "
Oct 30 22:51:43.038750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038308    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-dshm\") pod \"d3f74f0b-774f-4b9f-ada4-165542267ef4\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") "
Oct 30 22:51:43.038750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038339    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86ecfcbe-143b-46ef-ac5f-fed921354ef6-tls-certs\" (UniqueName: \"kubernetes.io/secret/86ecfcbe-143b-46ef-ac5f-fed921354ef6-tls-certs\") pod \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") "
Oct 30 22:51:43.038750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038337    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-model-cache" pod "d3f74f0b-774f-4b9f-ada4-165542267ef4" (UID: "d3f74f0b-774f-4b9f-ada4-165542267ef4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:43.038750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038354    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-model-cache" pod "86ecfcbe-143b-46ef-ac5f-fed921354ef6" (UID: "86ecfcbe-143b-46ef-ac5f-fed921354ef6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:43.038750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038372    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-tmp-dir\") pod \"d3f74f0b-774f-4b9f-ada4-165542267ef4\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") "
Oct 30 22:51:43.038750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038416    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-home\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-home\") pod \"d3f74f0b-774f-4b9f-ada4-165542267ef4\" (UID: \"d3f74f0b-774f-4b9f-ada4-165542267ef4\") "
Oct 30 22:51:43.038750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038445    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-model-cache\") pod \"4925f432-7a20-48b1-a052-d8ca2fd156e8\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") "
Oct 30 22:51:43.038750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038510    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-dshm\") pod \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\" (UID: \"86ecfcbe-143b-46ef-ac5f-fed921354ef6\") "
Oct 30 22:51:43.038750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.038574    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-dshm\") pod \"4925f432-7a20-48b1-a052-d8ca2fd156e8\" (UID: \"4925f432-7a20-48b1-a052-d8ca2fd156e8\") "
Oct 30 22:51:43.039338 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.039067    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-home" pod "d3f74f0b-774f-4b9f-ada4-165542267ef4" (UID: "d3f74f0b-774f-4b9f-ada4-165542267ef4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:43.039338 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.039277    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-model-cache" pod "4925f432-7a20-48b1-a052-d8ca2fd156e8" (UID: "4925f432-7a20-48b1-a052-d8ca2fd156e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:43.039449 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.039384    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-tmp-dir" pod "d3f74f0b-774f-4b9f-ada4-165542267ef4" (UID: "d3f74f0b-774f-4b9f-ada4-165542267ef4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:43.039982 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.039961    4553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-home\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.040092 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.039991    4553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-model-cache\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.040092 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.040005    4553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-model-cache\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.040092 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.040018    4553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-tmp-dir\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.040092 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.040048    4553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-home\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.040092 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.040060    4553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-model-cache\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.040092 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.040073    4553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-tmp-dir\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.040092 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.040086    4553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-tmp-dir\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.040710 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.040099    4553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-home\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.041386 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.041358    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f74f0b-774f-4b9f-ada4-165542267ef4-tls-certs" pod "d3f74f0b-774f-4b9f-ada4-165542267ef4" (UID: "d3f74f0b-774f-4b9f-ada4-165542267ef4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:43.041720 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.041689    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4925f432-7a20-48b1-a052-d8ca2fd156e8-tls-certs" pod "4925f432-7a20-48b1-a052-d8ca2fd156e8" (UID: "4925f432-7a20-48b1-a052-d8ca2fd156e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:43.041814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.041735    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ecfcbe-143b-46ef-ac5f-fed921354ef6-tls-certs" pod "86ecfcbe-143b-46ef-ac5f-fed921354ef6" (UID: "86ecfcbe-143b-46ef-ac5f-fed921354ef6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:43.041814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.041772    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4925f432-7a20-48b1-a052-d8ca2fd156e8-kube-api-access-w8ldh" pod "4925f432-7a20-48b1-a052-d8ca2fd156e8" (UID: "4925f432-7a20-48b1-a052-d8ca2fd156e8"). InnerVolumeSpecName "kube-api-access-w8ldh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:43.041909 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.041859    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ecfcbe-143b-46ef-ac5f-fed921354ef6-kube-api-access-phsqq" pod "86ecfcbe-143b-46ef-ac5f-fed921354ef6" (UID: "86ecfcbe-143b-46ef-ac5f-fed921354ef6"). InnerVolumeSpecName "kube-api-access-phsqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:43.041943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.041903    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-dshm" pod "4925f432-7a20-48b1-a052-d8ca2fd156e8" (UID: "4925f432-7a20-48b1-a052-d8ca2fd156e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:43.042346 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.042324    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f74f0b-774f-4b9f-ada4-165542267ef4-kube-api-access-5vjwk" pod "d3f74f0b-774f-4b9f-ada4-165542267ef4" (UID: "d3f74f0b-774f-4b9f-ada4-165542267ef4"). InnerVolumeSpecName "kube-api-access-5vjwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:43.042610 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.042591    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-dshm" pod "d3f74f0b-774f-4b9f-ada4-165542267ef4" (UID: "d3f74f0b-774f-4b9f-ada4-165542267ef4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:43.042610 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.042596    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-dshm" pod "86ecfcbe-143b-46ef-ac5f-fed921354ef6" (UID: "86ecfcbe-143b-46ef-ac5f-fed921354ef6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:43.141643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.141604    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phsqq\" (UniqueName: \"kubernetes.io/projected/86ecfcbe-143b-46ef-ac5f-fed921354ef6-kube-api-access-phsqq\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.141643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.141638    4553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4925f432-7a20-48b1-a052-d8ca2fd156e8-tls-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.141643 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.141652    4553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f74f0b-774f-4b9f-ada4-165542267ef4-tls-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.141910 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.141664    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8ldh\" (UniqueName: \"kubernetes.io/projected/4925f432-7a20-48b1-a052-d8ca2fd156e8-kube-api-access-w8ldh\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.141910 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.141676    4553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f74f0b-774f-4b9f-ada4-165542267ef4-dshm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.141910 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.141689    4553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ecfcbe-143b-46ef-ac5f-fed921354ef6-tls-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.141910 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.141699    4553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ecfcbe-143b-46ef-ac5f-fed921354ef6-dshm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.141910 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.141710    4553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4925f432-7a20-48b1-a052-d8ca2fd156e8-dshm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.141910 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.141723    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vjwk\" (UniqueName: \"kubernetes.io/projected/d3f74f0b-774f-4b9f-ada4-165542267ef4-kube-api-access-5vjwk\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:43.464272 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.464185    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"]
Oct 30 22:51:43.470699 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.470657    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf"]
Oct 30 22:51:43.580759 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.580726    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"]
Oct 30 22:51:43.587237 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.587204    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/simulator-w1-1c3eef-kserve-7d5d55dbf-8zb89"]
Oct 30 22:51:43.696740 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.696704    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"]
Oct 30 22:51:43.700284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.700256    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1c3eef/distinct-w1-1c3eef-kserve-8cc9bfc87-2spx9"]
Oct 30 22:51:43.906816 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:43.906763    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1c3eef/distinct-2-w1-1c3eef-kserve-cc494dfb-drjmf" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:51:44.853493 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:44.853450    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q" podUID="8a3d82a4-6522-40f9-a000-e92b7b95aa76" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:44.853890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:44.853538    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t" podUID="417967ed-ef2e-449f-95fe-fadf4e6926e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:45.393084 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:45.393050    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" path="/var/lib/kubelet/pods/4925f432-7a20-48b1-a052-d8ca2fd156e8/volumes"
Oct 30 22:51:45.393536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:45.393522    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" path="/var/lib/kubelet/pods/86ecfcbe-143b-46ef-ac5f-fed921354ef6/volumes"
Oct 30 22:51:45.393890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:45.393878    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" path="/var/lib/kubelet/pods/d3f74f0b-774f-4b9f-ada4-165542267ef4/volumes"
Oct 30 22:51:54.853915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:54.853878    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q" podUID="8a3d82a4-6522-40f9-a000-e92b7b95aa76" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:54.854352 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:54.853885    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t" podUID="417967ed-ef2e-449f-95fe-fadf4e6926e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:56.304826 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:56.304744    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"
Oct 30 22:51:56.398050 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:56.398018    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417967ed-ef2e-449f-95fe-fadf4e6926e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/417967ed-ef2e-449f-95fe-fadf4e6926e5-plugins-config-volume\") pod \"417967ed-ef2e-449f-95fe-fadf4e6926e5\" (UID: \"417967ed-ef2e-449f-95fe-fadf4e6926e5\") "
Oct 30 22:51:56.398332 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:56.398070    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417967ed-ef2e-449f-95fe-fadf4e6926e5-kube-api-access-tjrk6\" (UniqueName: \"kubernetes.io/projected/417967ed-ef2e-449f-95fe-fadf4e6926e5-kube-api-access-tjrk6\") pod \"417967ed-ef2e-449f-95fe-fadf4e6926e5\" (UID: \"417967ed-ef2e-449f-95fe-fadf4e6926e5\") "
Oct 30 22:51:56.398536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:56.398505    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417967ed-ef2e-449f-95fe-fadf4e6926e5-plugins-config-volume" pod "417967ed-ef2e-449f-95fe-fadf4e6926e5" (UID: "417967ed-ef2e-449f-95fe-fadf4e6926e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:56.400436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:56.400415    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417967ed-ef2e-449f-95fe-fadf4e6926e5-kube-api-access-tjrk6" pod "417967ed-ef2e-449f-95fe-fadf4e6926e5" (UID: "417967ed-ef2e-449f-95fe-fadf4e6926e5"). InnerVolumeSpecName "kube-api-access-tjrk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:56.420116 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:56.420086    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"
Oct 30 22:51:56.499061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:56.499032    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3d82a4-6522-40f9-a000-e92b7b95aa76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3d82a4-6522-40f9-a000-e92b7b95aa76-plugins-config-volume\") pod \"8a3d82a4-6522-40f9-a000-e92b7b95aa76\" (UID: \"8a3d82a4-6522-40f9-a000-e92b7b95aa76\") "
Oct 30 22:51:56.499061 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:56.499068    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3d82a4-6522-40f9-a000-e92b7b95aa76-kube-api-access-z7s6t\" (UniqueName: \"kubernetes.io/projected/8a3d82a4-6522-40f9-a000-e92b7b95aa76-kube-api-access-z7s6t\") pod \"8a3d82a4-6522-40f9-a000-e92b7b95aa76\" (UID: \"8a3d82a4-6522-40f9-a000-e92b7b95aa76\") "
Oct 30 22:51:56.499437 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:56.499414    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3d82a4-6522-40f9-a000-e92b7b95aa76-plugins-config-volume" pod "8a3d82a4-6522-40f9-a000-e92b7b95aa76" (UID: "8a3d82a4-6522-40f9-a000-e92b7b95aa76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:56.499601 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:56.499582    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/417967ed-ef2e-449f-95fe-fadf4e6926e5-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:56.499601 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:56.499601    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjrk6\" (UniqueName: \"kubernetes.io/projected/417967ed-ef2e-449f-95fe-fadf4e6926e5-kube-api-access-tjrk6\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:56.499776 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:56.499611    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3d82a4-6522-40f9-a000-e92b7b95aa76-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:56.501435 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:56.501407    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3d82a4-6522-40f9-a000-e92b7b95aa76-kube-api-access-z7s6t" pod "8a3d82a4-6522-40f9-a000-e92b7b95aa76" (UID: "8a3d82a4-6522-40f9-a000-e92b7b95aa76"). InnerVolumeSpecName "kube-api-access-z7s6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:56.600812 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:56.600780    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7s6t\" (UniqueName: \"kubernetes.io/projected/8a3d82a4-6522-40f9-a000-e92b7b95aa76-kube-api-access-z7s6t\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:51:57.065883 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.065854    4553 generic.go:356] "Generic (PLEG): container finished" podID="8a3d82a4-6522-40f9-a000-e92b7b95aa76" containerID="de16a9c2f1639636b15130b2e6af76bf9b6ef4edb98b82e0efbcc5ce9c2de3d0" exitCode=0
Oct 30 22:51:57.066040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.065920    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"
Oct 30 22:51:57.066040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.065927    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q" event={"ID":"8a3d82a4-6522-40f9-a000-e92b7b95aa76","Type":"ContainerDied","Data":"de16a9c2f1639636b15130b2e6af76bf9b6ef4edb98b82e0efbcc5ce9c2de3d0"}
Oct 30 22:51:57.066040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.066004    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q" event={"ID":"8a3d82a4-6522-40f9-a000-e92b7b95aa76","Type":"ContainerDied","Data":"84af5e95940b170023ce99dfee00293389d50ce162828b64a2697fd0652b5ef1"}
Oct 30 22:51:57.066040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.066026    4553 scope.go:122] "RemoveContainer" containerID="de16a9c2f1639636b15130b2e6af76bf9b6ef4edb98b82e0efbcc5ce9c2de3d0"
Oct 30 22:51:57.067704 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.067683    4553 generic.go:356] "Generic (PLEG): container finished" podID="417967ed-ef2e-449f-95fe-fadf4e6926e5" containerID="fc50e97c7b83bf97fe00fee61b59d0ca938a37ea0655cd5823fa6a8584db2500" exitCode=0
Oct 30 22:51:57.067814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.067737    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"
Oct 30 22:51:57.067876 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.067738    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t" event={"ID":"417967ed-ef2e-449f-95fe-fadf4e6926e5","Type":"ContainerDied","Data":"fc50e97c7b83bf97fe00fee61b59d0ca938a37ea0655cd5823fa6a8584db2500"}
Oct 30 22:51:57.067876 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.067863    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t" event={"ID":"417967ed-ef2e-449f-95fe-fadf4e6926e5","Type":"ContainerDied","Data":"89f2d5be915a25c91de29a42cd90b4e65f568086f2c8012f285e71c4a54cf95c"}
Oct 30 22:51:57.077661 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.077639    4553 scope.go:122] "RemoveContainer" containerID="de16a9c2f1639636b15130b2e6af76bf9b6ef4edb98b82e0efbcc5ce9c2de3d0"
Oct 30 22:51:57.078050 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:51:57.078023    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de16a9c2f1639636b15130b2e6af76bf9b6ef4edb98b82e0efbcc5ce9c2de3d0\": container with ID starting with de16a9c2f1639636b15130b2e6af76bf9b6ef4edb98b82e0efbcc5ce9c2de3d0 not found: ID does not exist" containerID="de16a9c2f1639636b15130b2e6af76bf9b6ef4edb98b82e0efbcc5ce9c2de3d0"
Oct 30 22:51:57.078168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.078062    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de16a9c2f1639636b15130b2e6af76bf9b6ef4edb98b82e0efbcc5ce9c2de3d0"} err="failed to get container status \"de16a9c2f1639636b15130b2e6af76bf9b6ef4edb98b82e0efbcc5ce9c2de3d0\": rpc error: code = NotFound desc = could not find container \"de16a9c2f1639636b15130b2e6af76bf9b6ef4edb98b82e0efbcc5ce9c2de3d0\": container with ID starting with de16a9c2f1639636b15130b2e6af76bf9b6ef4edb98b82e0efbcc5ce9c2de3d0 not found: ID does not exist"
Oct 30 22:51:57.078168 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.078086    4553 scope.go:122] "RemoveContainer" containerID="fc50e97c7b83bf97fe00fee61b59d0ca938a37ea0655cd5823fa6a8584db2500"
Oct 30 22:51:57.087338 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.087317    4553 scope.go:122] "RemoveContainer" containerID="fc50e97c7b83bf97fe00fee61b59d0ca938a37ea0655cd5823fa6a8584db2500"
Oct 30 22:51:57.087629 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:51:57.087607    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc50e97c7b83bf97fe00fee61b59d0ca938a37ea0655cd5823fa6a8584db2500\": container with ID starting with fc50e97c7b83bf97fe00fee61b59d0ca938a37ea0655cd5823fa6a8584db2500 not found: ID does not exist" containerID="fc50e97c7b83bf97fe00fee61b59d0ca938a37ea0655cd5823fa6a8584db2500"
Oct 30 22:51:57.087696 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.087638    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc50e97c7b83bf97fe00fee61b59d0ca938a37ea0655cd5823fa6a8584db2500"} err="failed to get container status \"fc50e97c7b83bf97fe00fee61b59d0ca938a37ea0655cd5823fa6a8584db2500\": rpc error: code = NotFound desc = could not find container \"fc50e97c7b83bf97fe00fee61b59d0ca938a37ea0655cd5823fa6a8584db2500\": container with ID starting with fc50e97c7b83bf97fe00fee61b59d0ca938a37ea0655cd5823fa6a8584db2500 not found: ID does not exist"
Oct 30 22:51:57.233237 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.233207    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"]
Oct 30 22:51:57.239330 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.239293    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e90906-df86d8647-ws84t"]
Oct 30 22:51:57.349851 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.349824    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"]
Oct 30 22:51:57.355813 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.355785    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e90906-647568c97f-mpn6q"]
Oct 30 22:51:57.393870 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.393839    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417967ed-ef2e-449f-95fe-fadf4e6926e5" path="/var/lib/kubelet/pods/417967ed-ef2e-449f-95fe-fadf4e6926e5/volumes"
Oct 30 22:51:57.394344 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:51:57.394330    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3d82a4-6522-40f9-a000-e92b7b95aa76" path="/var/lib/kubelet/pods/8a3d82a4-6522-40f9-a000-e92b7b95aa76/volumes"
Oct 30 22:51:58.201637 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:51:58.201601    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:52:03.508866 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:03.508820    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"]
Oct 30 22:52:03.509411 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:03.509088    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7" podUID="71914545-402c-416e-b1a8-99f17216e7fe" containerName="maas-api" containerID="cri-o://ef47eb06da3859d316d355bf6f783a4c91fcc73459e24aab3b218ebb5b495250" gracePeriod=30
Oct 30 22:52:03.763184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:03.763077    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:52:03.884075 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:03.884038    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71914545-402c-416e-b1a8-99f17216e7fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71914545-402c-416e-b1a8-99f17216e7fe-openshift-service-ca\") pod \"71914545-402c-416e-b1a8-99f17216e7fe\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") "
Oct 30 22:52:03.884304 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:03.884105    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71914545-402c-416e-b1a8-99f17216e7fe-kube-api-access-r9rq8\" (UniqueName: \"kubernetes.io/projected/71914545-402c-416e-b1a8-99f17216e7fe-kube-api-access-r9rq8\") pod \"71914545-402c-416e-b1a8-99f17216e7fe\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") "
Oct 30 22:52:03.884304 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:03.884184    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-maas-api-tls\") pod \"71914545-402c-416e-b1a8-99f17216e7fe\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") "
Oct 30 22:52:03.884304 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:03.884259    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-metrics-certs\") pod \"71914545-402c-416e-b1a8-99f17216e7fe\" (UID: \"71914545-402c-416e-b1a8-99f17216e7fe\") "
Oct 30 22:52:03.889368 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:03.889334    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71914545-402c-416e-b1a8-99f17216e7fe-openshift-service-ca" pod "71914545-402c-416e-b1a8-99f17216e7fe" (UID: "71914545-402c-416e-b1a8-99f17216e7fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:03.897174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:03.897102    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-maas-api-tls" pod "71914545-402c-416e-b1a8-99f17216e7fe" (UID: "71914545-402c-416e-b1a8-99f17216e7fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:03.897174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:03.897121    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-metrics-certs" pod "71914545-402c-416e-b1a8-99f17216e7fe" (UID: "71914545-402c-416e-b1a8-99f17216e7fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:03.897174 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:03.897125    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71914545-402c-416e-b1a8-99f17216e7fe-kube-api-access-r9rq8" pod "71914545-402c-416e-b1a8-99f17216e7fe" (UID: "71914545-402c-416e-b1a8-99f17216e7fe"). InnerVolumeSpecName "kube-api-access-r9rq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:03.986087 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:03.986049    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71914545-402c-416e-b1a8-99f17216e7fe-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:03.986087 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:03.986081    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9rq8\" (UniqueName: \"kubernetes.io/projected/71914545-402c-416e-b1a8-99f17216e7fe-kube-api-access-r9rq8\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:03.986087 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:03.986094    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:03.986383 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:03.986103    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71914545-402c-416e-b1a8-99f17216e7fe-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:04.124510 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:04.124471    4553 generic.go:356] "Generic (PLEG): container finished" podID="71914545-402c-416e-b1a8-99f17216e7fe" containerID="ef47eb06da3859d316d355bf6f783a4c91fcc73459e24aab3b218ebb5b495250" exitCode=0
Oct 30 22:52:04.124699 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:04.124525    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"
Oct 30 22:52:04.124699 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:04.124538    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7" event={"ID":"71914545-402c-416e-b1a8-99f17216e7fe","Type":"ContainerDied","Data":"ef47eb06da3859d316d355bf6f783a4c91fcc73459e24aab3b218ebb5b495250"}
Oct 30 22:52:04.124699 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:04.124586    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7" event={"ID":"71914545-402c-416e-b1a8-99f17216e7fe","Type":"ContainerDied","Data":"cb0b4b046062ac00ff031ddade9609f1dd7b10590a57401bdce2ef761304d980"}
Oct 30 22:52:04.124699 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:04.124608    4553 scope.go:122] "RemoveContainer" containerID="ef47eb06da3859d316d355bf6f783a4c91fcc73459e24aab3b218ebb5b495250"
Oct 30 22:52:04.136252 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:04.136226    4553 scope.go:122] "RemoveContainer" containerID="ef47eb06da3859d316d355bf6f783a4c91fcc73459e24aab3b218ebb5b495250"
Oct 30 22:52:04.136658 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:52:04.136629    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef47eb06da3859d316d355bf6f783a4c91fcc73459e24aab3b218ebb5b495250\": container with ID starting with ef47eb06da3859d316d355bf6f783a4c91fcc73459e24aab3b218ebb5b495250 not found: ID does not exist" containerID="ef47eb06da3859d316d355bf6f783a4c91fcc73459e24aab3b218ebb5b495250"
Oct 30 22:52:04.136929 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:04.136669    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef47eb06da3859d316d355bf6f783a4c91fcc73459e24aab3b218ebb5b495250"} err="failed to get container status \"ef47eb06da3859d316d355bf6f783a4c91fcc73459e24aab3b218ebb5b495250\": rpc error: code = NotFound desc = could not find container \"ef47eb06da3859d316d355bf6f783a4c91fcc73459e24aab3b218ebb5b495250\": container with ID starting with ef47eb06da3859d316d355bf6f783a4c91fcc73459e24aab3b218ebb5b495250 not found: ID does not exist"
Oct 30 22:52:04.291272 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:04.291234    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"]
Oct 30 22:52:04.294905 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:04.294879    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e90906-748bfd7494-b8bh7"]
Oct 30 22:52:05.393411 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:05.393375    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71914545-402c-416e-b1a8-99f17216e7fe" path="/var/lib/kubelet/pods/71914545-402c-416e-b1a8-99f17216e7fe/volumes"
Oct 30 22:52:13.770435 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:13.770357    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"]
Oct 30 22:52:13.770874 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:13.770624    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4" podUID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerName="istio-proxy" containerID="cri-o://d0be27e66cb690257f3c0a03762b7c42b36de707ac285ad43d66364def4f4db0" gracePeriod=30
Oct 30 22:52:16.896070 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:16.896030    4553 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:52:16.896578 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:16.896106    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4" podUID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:52:19.041426 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.041340    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:52:19.159284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.159250    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-workload-socket\") pod \"c73136ea-01b3-4ded-ab76-1c902be19f3a\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") "
Oct 30 22:52:19.159500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.159290    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-credential-socket\") pod \"c73136ea-01b3-4ded-ab76-1c902be19f3a\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") "
Oct 30 22:52:19.159500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.159367    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-token\" (UniqueName: \"kubernetes.io/projected/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-token\") pod \"c73136ea-01b3-4ded-ab76-1c902be19f3a\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") "
Oct 30 22:52:19.159500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.159418    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-envoy\") pod \"c73136ea-01b3-4ded-ab76-1c902be19f3a\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") "
Oct 30 22:52:19.159500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.159437    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-workload-certs\") pod \"c73136ea-01b3-4ded-ab76-1c902be19f3a\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") "
Oct 30 22:52:19.159500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.159478    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c73136ea-01b3-4ded-ab76-1c902be19f3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73136ea-01b3-4ded-ab76-1c902be19f3a-istiod-ca-cert\") pod \"c73136ea-01b3-4ded-ab76-1c902be19f3a\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") "
Oct 30 22:52:19.159904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.159599    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73136ea-01b3-4ded-ab76-1c902be19f3a-kube-api-access-fz447\" (UniqueName: \"kubernetes.io/projected/c73136ea-01b3-4ded-ab76-1c902be19f3a-kube-api-access-fz447\") pod \"c73136ea-01b3-4ded-ab76-1c902be19f3a\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") "
Oct 30 22:52:19.159904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.159603    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-workload-socket" pod "c73136ea-01b3-4ded-ab76-1c902be19f3a" (UID: "c73136ea-01b3-4ded-ab76-1c902be19f3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:19.159904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.159666    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-credential-socket" pod "c73136ea-01b3-4ded-ab76-1c902be19f3a" (UID: "c73136ea-01b3-4ded-ab76-1c902be19f3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:19.159904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.159720    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-data\") pod \"c73136ea-01b3-4ded-ab76-1c902be19f3a\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") "
Oct 30 22:52:19.159904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.159766    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-podinfo\") pod \"c73136ea-01b3-4ded-ab76-1c902be19f3a\" (UID: \"c73136ea-01b3-4ded-ab76-1c902be19f3a\") "
Oct 30 22:52:19.159904 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.159800    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-workload-certs" pod "c73136ea-01b3-4ded-ab76-1c902be19f3a" (UID: "c73136ea-01b3-4ded-ab76-1c902be19f3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:19.160236 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.159996    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-data" pod "c73136ea-01b3-4ded-ab76-1c902be19f3a" (UID: "c73136ea-01b3-4ded-ab76-1c902be19f3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:19.160236 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.160041    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73136ea-01b3-4ded-ab76-1c902be19f3a-istiod-ca-cert" pod "c73136ea-01b3-4ded-ab76-1c902be19f3a" (UID: "c73136ea-01b3-4ded-ab76-1c902be19f3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:19.160504 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.160483    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:19.160577 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.160510    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73136ea-01b3-4ded-ab76-1c902be19f3a-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:19.160577 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.160527    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:19.160577 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.160540    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:19.160577 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.160554    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:19.161873 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.161847    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-envoy" pod "c73136ea-01b3-4ded-ab76-1c902be19f3a" (UID: "c73136ea-01b3-4ded-ab76-1c902be19f3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:19.162076 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.162057    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-podinfo" pod "c73136ea-01b3-4ded-ab76-1c902be19f3a" (UID: "c73136ea-01b3-4ded-ab76-1c902be19f3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:52:19.162449 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.162430    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73136ea-01b3-4ded-ab76-1c902be19f3a-kube-api-access-fz447" pod "c73136ea-01b3-4ded-ab76-1c902be19f3a" (UID: "c73136ea-01b3-4ded-ab76-1c902be19f3a"). InnerVolumeSpecName "kube-api-access-fz447". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:19.162449 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.162439    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-token" pod "c73136ea-01b3-4ded-ab76-1c902be19f3a" (UID: "c73136ea-01b3-4ded-ab76-1c902be19f3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:19.217588 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.217557    4553 generic.go:356] "Generic (PLEG): container finished" podID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerID="d0be27e66cb690257f3c0a03762b7c42b36de707ac285ad43d66364def4f4db0" exitCode=0
Oct 30 22:52:19.217790 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.217614    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"
Oct 30 22:52:19.217790 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.217632    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4" event={"ID":"c73136ea-01b3-4ded-ab76-1c902be19f3a","Type":"ContainerDied","Data":"d0be27e66cb690257f3c0a03762b7c42b36de707ac285ad43d66364def4f4db0"}
Oct 30 22:52:19.217790 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.217686    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4" event={"ID":"c73136ea-01b3-4ded-ab76-1c902be19f3a","Type":"ContainerDied","Data":"e74d012f96ca7ee695b32716caedc8d96a7d200284ba562b5e80527efee90902"}
Oct 30 22:52:19.217790 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.217708    4553 scope.go:122] "RemoveContainer" containerID="d0be27e66cb690257f3c0a03762b7c42b36de707ac285ad43d66364def4f4db0"
Oct 30 22:52:19.228287 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.228268    4553 scope.go:122] "RemoveContainer" containerID="f39cb08c1b8b9a973b90a03215655c7a8a7d0ba8f4e91cc85305315418e0f6f9"
Oct 30 22:52:19.237211 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.237194    4553 scope.go:122] "RemoveContainer" containerID="d0be27e66cb690257f3c0a03762b7c42b36de707ac285ad43d66364def4f4db0"
Oct 30 22:52:19.237470 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:52:19.237454    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0be27e66cb690257f3c0a03762b7c42b36de707ac285ad43d66364def4f4db0\": container with ID starting with d0be27e66cb690257f3c0a03762b7c42b36de707ac285ad43d66364def4f4db0 not found: ID does not exist" containerID="d0be27e66cb690257f3c0a03762b7c42b36de707ac285ad43d66364def4f4db0"
Oct 30 22:52:19.237523 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.237480    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0be27e66cb690257f3c0a03762b7c42b36de707ac285ad43d66364def4f4db0"} err="failed to get container status \"d0be27e66cb690257f3c0a03762b7c42b36de707ac285ad43d66364def4f4db0\": rpc error: code = NotFound desc = could not find container \"d0be27e66cb690257f3c0a03762b7c42b36de707ac285ad43d66364def4f4db0\": container with ID starting with d0be27e66cb690257f3c0a03762b7c42b36de707ac285ad43d66364def4f4db0 not found: ID does not exist"
Oct 30 22:52:19.237523 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.237506    4553 scope.go:122] "RemoveContainer" containerID="f39cb08c1b8b9a973b90a03215655c7a8a7d0ba8f4e91cc85305315418e0f6f9"
Oct 30 22:52:19.237744 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:52:19.237726    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39cb08c1b8b9a973b90a03215655c7a8a7d0ba8f4e91cc85305315418e0f6f9\": container with ID starting with f39cb08c1b8b9a973b90a03215655c7a8a7d0ba8f4e91cc85305315418e0f6f9 not found: ID does not exist" containerID="f39cb08c1b8b9a973b90a03215655c7a8a7d0ba8f4e91cc85305315418e0f6f9"
Oct 30 22:52:19.237786 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.237749    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39cb08c1b8b9a973b90a03215655c7a8a7d0ba8f4e91cc85305315418e0f6f9"} err="failed to get container status \"f39cb08c1b8b9a973b90a03215655c7a8a7d0ba8f4e91cc85305315418e0f6f9\": rpc error: code = NotFound desc = could not find container \"f39cb08c1b8b9a973b90a03215655c7a8a7d0ba8f4e91cc85305315418e0f6f9\": container with ID starting with f39cb08c1b8b9a973b90a03215655c7a8a7d0ba8f4e91cc85305315418e0f6f9 not found: ID does not exist"
Oct 30 22:52:19.262019 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.261992    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz447\" (UniqueName: \"kubernetes.io/projected/c73136ea-01b3-4ded-ab76-1c902be19f3a-kube-api-access-fz447\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:19.262019 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.262017    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:19.262243 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.262026    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:19.262243 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.262036    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73136ea-01b3-4ded-ab76-1c902be19f3a-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:19.353255 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.353221    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"]
Oct 30 22:52:19.353876 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.353857    4553 cpu_manager.go:414] "containerMap: removing container" podUID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerName="istio-proxy"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.353878    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerName="istio-proxy"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.353895    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d8da6201-f247-4182-a143-6a880f98b175" containerName="istio-proxy"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.353903    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8da6201-f247-4182-a143-6a880f98b175" containerName="istio-proxy"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.353922    4553 cpu_manager.go:414] "containerMap: removing container" podUID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerName="istio-proxy"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.353929    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e1ba27-d171-44ac-b625-2d2e34fb1481" containerName="istio-proxy"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.353943    4553 cpu_manager.go:414] "containerMap: removing container" podUID="417967ed-ef2e-449f-95fe-fadf4e6926e5" containerName="payload-processing"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.353951    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417967ed-ef2e-449f-95fe-fadf4e6926e5" containerName="payload-processing"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.353969    4553 cpu_manager.go:414] "containerMap: removing container" podUID="f0886f20-1c73-4f9a-8f1d-63db1a2c5aba" containerName="payload-pre-processing"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.353976    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0886f20-1c73-4f9a-8f1d-63db1a2c5aba" containerName="payload-pre-processing"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.353986    4553 cpu_manager.go:414] "containerMap: removing container" podUID="8a3d82a4-6522-40f9-a000-e92b7b95aa76" containerName="payload-pre-processing"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.353995    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3d82a4-6522-40f9-a000-e92b7b95aa76" containerName="payload-pre-processing"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354012    4553 cpu_manager.go:414] "containerMap: removing container" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354019    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354034    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354040    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354053    4553 cpu_manager.go:414] "containerMap: removing container" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main"
Oct 30 22:52:19.354059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354060    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354071    4553 cpu_manager.go:414] "containerMap: removing container" podUID="b4a2de90-2ce9-452e-a5ed-63606a25e513" containerName="istio-proxy"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354077    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a2de90-2ce9-452e-a5ed-63606a25e513" containerName="istio-proxy"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354095    4553 cpu_manager.go:414] "containerMap: removing container" podUID="b4a2de90-2ce9-452e-a5ed-63606a25e513" containerName="istio-proxy"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354102    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a2de90-2ce9-452e-a5ed-63606a25e513" containerName="istio-proxy"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354112    4553 cpu_manager.go:414] "containerMap: removing container" podUID="310c1cdc-3c1b-4d84-9f50-9ac5e59e6198" containerName="payload-processing"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354119    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="310c1cdc-3c1b-4d84-9f50-9ac5e59e6198" containerName="payload-processing"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354171    4553 cpu_manager.go:414] "containerMap: removing container" podUID="71914545-402c-416e-b1a8-99f17216e7fe" containerName="maas-api"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354179    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71914545-402c-416e-b1a8-99f17216e7fe" containerName="maas-api"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354327    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a2de90-2ce9-452e-a5ed-63606a25e513" containerName="istio-proxy"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354340    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="310c1cdc-3c1b-4d84-9f50-9ac5e59e6198" containerName="payload-processing"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354349    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerName="istio-proxy"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354361    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8da6201-f247-4182-a143-6a880f98b175" containerName="istio-proxy"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354373    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="71914545-402c-416e-b1a8-99f17216e7fe" containerName="maas-api"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354383    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8da6201-f247-4182-a143-6a880f98b175" containerName="istio-proxy"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354395    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="417967ed-ef2e-449f-95fe-fadf4e6926e5" containerName="payload-processing"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354406    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0886f20-1c73-4f9a-8f1d-63db1a2c5aba" containerName="payload-pre-processing"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354418    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3d82a4-6522-40f9-a000-e92b7b95aa76" containerName="payload-pre-processing"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354429    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ecfcbe-143b-46ef-ac5f-fed921354ef6" containerName="main"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354440    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f74f0b-774f-4b9f-ada4-165542267ef4" containerName="main"
Oct 30 22:52:19.354931 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.354450    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="4925f432-7a20-48b1-a052-d8ca2fd156e8" containerName="main"
Oct 30 22:52:19.367890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.367856    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.400124 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.400090    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"]
Oct 30 22:52:19.403723 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.403690    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e90906-openshift-default-57cd56f465-6fcf4"]
Oct 30 22:52:19.421084 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.421057    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-3faf84-openshift-default-dockercfg-m96bp\""
Oct 30 22:52:19.421874 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.421853    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"]
Oct 30 22:52:19.464991 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.464959    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-token\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.465200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.465078    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-workload-certs\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.465200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.465120    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nvnh\" (UniqueName: \"kubernetes.io/projected/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-kube-api-access-8nvnh\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.465295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.465204    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-credential-socket\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.465295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.465246    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-envoy\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.465363 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.465329    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-podinfo\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.465402 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.465384    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istiod-ca-cert\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.465436 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.465424    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-workload-socket\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.465472 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.465449    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-data\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.566417 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.566385    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-podinfo\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.566628 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.566430    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istiod-ca-cert\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.566628 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.566447    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-workload-socket\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.566628 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.566486    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-data\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.566628 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.566534    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-token\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.566628 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.566619    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-workload-certs\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.566898 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.566688    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nvnh\" (UniqueName: \"kubernetes.io/projected/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-kube-api-access-8nvnh\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.566898 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.566718    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-credential-socket\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.567006 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.566919    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-workload-socket\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.567062 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.567000    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-envoy\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.567193 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.567170    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-workload-certs\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.567259 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.567224    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-data\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.567259 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.567247    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-credential-socket\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.567709 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.567616    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istiod-ca-cert\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.569763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.569727    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-podinfo\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.569763 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.569746    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-envoy\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.625416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.624722    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-token\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.625582 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.625436    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nvnh\" (UniqueName: \"kubernetes.io/projected/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-kube-api-access-8nvnh\") pod \"e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") " pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:19.684648 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:19.684609    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:20.119469 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:20.119434    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"]
Oct 30 22:52:20.120728 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:52:20.120702    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d1f1e4c_192c_4eb9_8f74_d9c21d31ad4b.slice/crio-ef117c040ac0dcd00433a38bcfbb5f6cf0508269385afb531f4588100313e80a WatchSource:0}: Error finding container ef117c040ac0dcd00433a38bcfbb5f6cf0508269385afb531f4588100313e80a: Status 404 returned error can't find the container with id ef117c040ac0dcd00433a38bcfbb5f6cf0508269385afb531f4588100313e80a
Oct 30 22:52:20.123293 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:20.123261    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:52:20.123389 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:20.123326    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:52:20.123389 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:20.123355    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:52:20.226793 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:20.226747    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh" event={"ID":"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b","Type":"ContainerStarted","Data":"3e70ceac94de7789efc6e17df2f2d46f661a842be52c3316f7017e9485b0e487"}
Oct 30 22:52:20.226976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:20.226827    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh" event={"ID":"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b","Type":"ContainerStarted","Data":"ef117c040ac0dcd00433a38bcfbb5f6cf0508269385afb531f4588100313e80a"}
Oct 30 22:52:20.685149 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:20.685103    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:21.393222 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:21.393190    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73136ea-01b3-4ded-ab76-1c902be19f3a" path="/var/lib/kubelet/pods/c73136ea-01b3-4ded-ab76-1c902be19f3a/volumes"
Oct 30 22:52:21.690610 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:21.690535    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:21.862227 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:21.862174    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh" podStartSLOduration=2.8621619320000002 podStartE2EDuration="2.862161932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:20.351414066 +0000 UTC m=+4191.085777097" watchObservedRunningTime="2026-10-30 22:52:21.862161932 +0000 UTC m=+4192.596524954"
Oct 30 22:52:22.240983 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:22.240953    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:22.242076 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:22.242054    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:52:23.848417 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:23.848380    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"]
Oct 30 22:52:26.238892 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.238854    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"]
Oct 30 22:52:26.291497 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.291443    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"]
Oct 30 22:52:26.292106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.291741    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl" podUID="fa41d436-167b-4135-93b5-7b090231e65b" containerName="maas-api" containerID="cri-o://30006317e26b1d7fe4f2eeb209daaf4138ae96a4aa7eb2dd2c3d8f5c2341534c" gracePeriod=30
Oct 30 22:52:26.337508 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:52:26.337474    4553 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa41d436_167b_4135_93b5_7b090231e65b.slice/crio-conmon-30006317e26b1d7fe4f2eeb209daaf4138ae96a4aa7eb2dd2c3d8f5c2341534c.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:52:26.337714 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:52:26.337565    4553 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa41d436_167b_4135_93b5_7b090231e65b.slice/crio-conmon-30006317e26b1d7fe4f2eeb209daaf4138ae96a4aa7eb2dd2c3d8f5c2341534c.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:52:26.355856 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.355827    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"]
Oct 30 22:52:26.570588 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.570559    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:52:26.646804 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.646764    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-metrics-certs\") pod \"fa41d436-167b-4135-93b5-7b090231e65b\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") "
Oct 30 22:52:26.646990 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.646827    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa41d436-167b-4135-93b5-7b090231e65b-kube-api-access-6w67z\" (UniqueName: \"kubernetes.io/projected/fa41d436-167b-4135-93b5-7b090231e65b-kube-api-access-6w67z\") pod \"fa41d436-167b-4135-93b5-7b090231e65b\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") "
Oct 30 22:52:26.646990 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.646866    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-maas-api-tls\") pod \"fa41d436-167b-4135-93b5-7b090231e65b\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") "
Oct 30 22:52:26.646990 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.646961    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa41d436-167b-4135-93b5-7b090231e65b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa41d436-167b-4135-93b5-7b090231e65b-openshift-service-ca\") pod \"fa41d436-167b-4135-93b5-7b090231e65b\" (UID: \"fa41d436-167b-4135-93b5-7b090231e65b\") "
Oct 30 22:52:26.647471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.647444    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa41d436-167b-4135-93b5-7b090231e65b-openshift-service-ca" pod "fa41d436-167b-4135-93b5-7b090231e65b" (UID: "fa41d436-167b-4135-93b5-7b090231e65b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:26.649188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.649163    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-metrics-certs" pod "fa41d436-167b-4135-93b5-7b090231e65b" (UID: "fa41d436-167b-4135-93b5-7b090231e65b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:26.649713 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.649688    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-maas-api-tls" pod "fa41d436-167b-4135-93b5-7b090231e65b" (UID: "fa41d436-167b-4135-93b5-7b090231e65b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:26.649804 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.649746    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa41d436-167b-4135-93b5-7b090231e65b-kube-api-access-6w67z" pod "fa41d436-167b-4135-93b5-7b090231e65b" (UID: "fa41d436-167b-4135-93b5-7b090231e65b"). InnerVolumeSpecName "kube-api-access-6w67z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:26.748186 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.748128    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:26.748186 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.748172    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w67z\" (UniqueName: \"kubernetes.io/projected/fa41d436-167b-4135-93b5-7b090231e65b-kube-api-access-6w67z\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:26.748186 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.748184    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa41d436-167b-4135-93b5-7b090231e65b-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:26.748186 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:26.748193    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa41d436-167b-4135-93b5-7b090231e65b-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:27.271298 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:27.271261    4553 generic.go:356] "Generic (PLEG): container finished" podID="fa41d436-167b-4135-93b5-7b090231e65b" containerID="30006317e26b1d7fe4f2eeb209daaf4138ae96a4aa7eb2dd2c3d8f5c2341534c" exitCode=0
Oct 30 22:52:27.271849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:27.271329    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"
Oct 30 22:52:27.271849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:27.271343    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl" event={"ID":"fa41d436-167b-4135-93b5-7b090231e65b","Type":"ContainerDied","Data":"30006317e26b1d7fe4f2eeb209daaf4138ae96a4aa7eb2dd2c3d8f5c2341534c"}
Oct 30 22:52:27.271849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:27.271426    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl" event={"ID":"fa41d436-167b-4135-93b5-7b090231e65b","Type":"ContainerDied","Data":"fcb756b5d1db126e3bd503c948b5693ac32b69314a3799650560aa4452eb40d3"}
Oct 30 22:52:27.271849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:27.271448    4553 scope.go:122] "RemoveContainer" containerID="30006317e26b1d7fe4f2eeb209daaf4138ae96a4aa7eb2dd2c3d8f5c2341534c"
Oct 30 22:52:27.283529 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:27.283506    4553 scope.go:122] "RemoveContainer" containerID="30006317e26b1d7fe4f2eeb209daaf4138ae96a4aa7eb2dd2c3d8f5c2341534c"
Oct 30 22:52:27.283835 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:52:27.283815    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30006317e26b1d7fe4f2eeb209daaf4138ae96a4aa7eb2dd2c3d8f5c2341534c\": container with ID starting with 30006317e26b1d7fe4f2eeb209daaf4138ae96a4aa7eb2dd2c3d8f5c2341534c not found: ID does not exist" containerID="30006317e26b1d7fe4f2eeb209daaf4138ae96a4aa7eb2dd2c3d8f5c2341534c"
Oct 30 22:52:27.283905 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:27.283848    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30006317e26b1d7fe4f2eeb209daaf4138ae96a4aa7eb2dd2c3d8f5c2341534c"} err="failed to get container status \"30006317e26b1d7fe4f2eeb209daaf4138ae96a4aa7eb2dd2c3d8f5c2341534c\": rpc error: code = NotFound desc = could not find container \"30006317e26b1d7fe4f2eeb209daaf4138ae96a4aa7eb2dd2c3d8f5c2341534c\": container with ID starting with 30006317e26b1d7fe4f2eeb209daaf4138ae96a4aa7eb2dd2c3d8f5c2341534c not found: ID does not exist"
Oct 30 22:52:27.438644 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:27.438613    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"]
Oct 30 22:52:27.443661 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:27.443628    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b1eb3f-57c9fc57ff-lbmnl"]
Oct 30 22:52:28.201885 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:52:28.201847    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:52:28.849375 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:28.849312    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq" podUID="7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932" containerName="payload-pre-processing" containerID="cri-o://55a2e1bd9d555415579910511eb9f17bc3dd425cd76995fa1ec9fd911c6eb7ac" gracePeriod=25
Oct 30 22:52:29.393747 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:29.393714    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa41d436-167b-4135-93b5-7b090231e65b" path="/var/lib/kubelet/pods/fa41d436-167b-4135-93b5-7b090231e65b/volumes"
Oct 30 22:52:30.089968 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:30.089941    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:52:30.101010 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:30.100987    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:52:30.660536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:30.660499    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq" podUID="7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:52:31.239314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:31.239265    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8" podUID="3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3" containerName="payload-processing" containerID="cri-o://4a87b50dc81c246b1b79f4949bf9e0662c49dcc88222ca114f12469c574aaccd" gracePeriod=25
Oct 30 22:52:31.357040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:31.356969    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq" podUID="289e89bb-9bee-47f8-8768-8ac8c35b47e5" containerName="payload-pre-processing" containerID="cri-o://def8e973ef186f5abf22374f8931585193301a7fd01af3b6656cfc40fae9804d" gracePeriod=25
Oct 30 22:52:39.741400 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:39.741350    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq" podUID="289e89bb-9bee-47f8-8768-8ac8c35b47e5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:52:39.742029 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:39.741350    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8" podUID="3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:52:40.659502 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:40.659465    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq" podUID="7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:52:43.406236 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:43.406199    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"]
Oct 30 22:52:43.456517 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:43.456486    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"]
Oct 30 22:52:46.849611 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.849567    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"]
Oct 30 22:52:46.850947 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.850922    4553 cpu_manager.go:414] "containerMap: removing container" podUID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerName="istio-proxy"
Oct 30 22:52:46.851158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.851120    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerName="istio-proxy"
Oct 30 22:52:46.851270 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.851255    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d8da6201-f247-4182-a143-6a880f98b175" containerName="istio-proxy"
Oct 30 22:52:46.851344 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.851271    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8da6201-f247-4182-a143-6a880f98b175" containerName="istio-proxy"
Oct 30 22:52:46.851344 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.851301    4553 cpu_manager.go:414] "containerMap: removing container" podUID="fa41d436-167b-4135-93b5-7b090231e65b" containerName="maas-api"
Oct 30 22:52:46.851344 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.851310    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa41d436-167b-4135-93b5-7b090231e65b" containerName="maas-api"
Oct 30 22:52:46.851497 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.851482    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa41d436-167b-4135-93b5-7b090231e65b" containerName="maas-api"
Oct 30 22:52:46.851549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.851506    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a2de90-2ce9-452e-a5ed-63606a25e513" containerName="istio-proxy"
Oct 30 22:52:46.851549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.851516    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73136ea-01b3-4ded-ab76-1c902be19f3a" containerName="istio-proxy"
Oct 30 22:52:46.854849 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.854826    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"
Oct 30 22:52:46.879918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.879886    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxxlm\" (UniqueName: \"kubernetes.io/projected/c065085c-3963-45a0-b4bc-09ad435e27b7-kube-api-access-dxxlm\") pod \"payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n\" (UID: \"c065085c-3963-45a0-b4bc-09ad435e27b7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"
Oct 30 22:52:46.880128 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.880104    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c065085c-3963-45a0-b4bc-09ad435e27b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n\" (UID: \"c065085c-3963-45a0-b4bc-09ad435e27b7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"
Oct 30 22:52:46.908273 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.908237    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"]
Oct 30 22:52:46.955835 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.955802    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-3faf84\""
Oct 30 22:52:46.981696 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.981659    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c065085c-3963-45a0-b4bc-09ad435e27b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n\" (UID: \"c065085c-3963-45a0-b4bc-09ad435e27b7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"
Oct 30 22:52:46.981860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.981724    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxxlm\" (UniqueName: \"kubernetes.io/projected/c065085c-3963-45a0-b4bc-09ad435e27b7-kube-api-access-dxxlm\") pod \"payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n\" (UID: \"c065085c-3963-45a0-b4bc-09ad435e27b7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"
Oct 30 22:52:46.982412 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.982393    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c065085c-3963-45a0-b4bc-09ad435e27b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n\" (UID: \"c065085c-3963-45a0-b4bc-09ad435e27b7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"
Oct 30 22:52:46.995732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.995690    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"]
Oct 30 22:52:46.999921 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:46.999897    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"
Oct 30 22:52:47.040431 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:47.040400    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxxlm\" (UniqueName: \"kubernetes.io/projected/c065085c-3963-45a0-b4bc-09ad435e27b7-kube-api-access-dxxlm\") pod \"payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n\" (UID: \"c065085c-3963-45a0-b4bc-09ad435e27b7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"
Oct 30 22:52:47.055302 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:47.055272    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-3faf84-dockercfg-465bj\""
Oct 30 22:52:47.059063 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:47.059027    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"]
Oct 30 22:52:47.082729 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:47.082679    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eaa3e7a-052e-4f6a-9cd3-d3a648528160-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r\" (UID: \"0eaa3e7a-052e-4f6a-9cd3-d3a648528160\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"
Oct 30 22:52:47.082933 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:47.082908    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf5sq\" (UniqueName: \"kubernetes.io/projected/0eaa3e7a-052e-4f6a-9cd3-d3a648528160-kube-api-access-nf5sq\") pod \"payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r\" (UID: \"0eaa3e7a-052e-4f6a-9cd3-d3a648528160\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"
Oct 30 22:52:47.171814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:47.171726    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"
Oct 30 22:52:47.184551 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:47.184524    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf5sq\" (UniqueName: \"kubernetes.io/projected/0eaa3e7a-052e-4f6a-9cd3-d3a648528160-kube-api-access-nf5sq\") pod \"payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r\" (UID: \"0eaa3e7a-052e-4f6a-9cd3-d3a648528160\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"
Oct 30 22:52:47.184690 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:47.184671    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eaa3e7a-052e-4f6a-9cd3-d3a648528160-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r\" (UID: \"0eaa3e7a-052e-4f6a-9cd3-d3a648528160\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"
Oct 30 22:52:47.185338 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:47.185318    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eaa3e7a-052e-4f6a-9cd3-d3a648528160-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r\" (UID: \"0eaa3e7a-052e-4f6a-9cd3-d3a648528160\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"
Oct 30 22:52:47.248567 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:47.248526    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf5sq\" (UniqueName: \"kubernetes.io/projected/0eaa3e7a-052e-4f6a-9cd3-d3a648528160-kube-api-access-nf5sq\") pod \"payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r\" (UID: \"0eaa3e7a-052e-4f6a-9cd3-d3a648528160\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"
Oct 30 22:52:47.314780 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:47.314386    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"
Oct 30 22:52:47.550853 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:47.550822    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"]
Oct 30 22:52:47.552168 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:52:47.552116    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc065085c_3963_45a0_b4bc_09ad435e27b7.slice/crio-cf650300e6dcc0a5ec750cfcc6505bc922bf486e44ee7adeea7cc726f7059b83 WatchSource:0}: Error finding container cf650300e6dcc0a5ec750cfcc6505bc922bf486e44ee7adeea7cc726f7059b83: Status 404 returned error can't find the container with id cf650300e6dcc0a5ec750cfcc6505bc922bf486e44ee7adeea7cc726f7059b83
Oct 30 22:52:47.693956 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:47.693922    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"]
Oct 30 22:52:47.695554 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:52:47.695526    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eaa3e7a_052e_4f6a_9cd3_d3a648528160.slice/crio-abf095c4a6012755cc1a0817bb1d46c150d407653f16c70dd43f7da9d1113511 WatchSource:0}: Error finding container abf095c4a6012755cc1a0817bb1d46c150d407653f16c70dd43f7da9d1113511: Status 404 returned error can't find the container with id abf095c4a6012755cc1a0817bb1d46c150d407653f16c70dd43f7da9d1113511
Oct 30 22:52:48.399976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:48.399913    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r" event={"ID":"0eaa3e7a-052e-4f6a-9cd3-d3a648528160","Type":"ContainerStarted","Data":"b47e3ac64d60033222669276d6870928f16f14c241c8b8772bcba1b5d16c1add"}
Oct 30 22:52:48.400530 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:48.400089    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"
Oct 30 22:52:48.400530 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:48.400169    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r" event={"ID":"0eaa3e7a-052e-4f6a-9cd3-d3a648528160","Type":"ContainerStarted","Data":"abf095c4a6012755cc1a0817bb1d46c150d407653f16c70dd43f7da9d1113511"}
Oct 30 22:52:48.401595 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:48.401571    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n" event={"ID":"c065085c-3963-45a0-b4bc-09ad435e27b7","Type":"ContainerStarted","Data":"dda1ba985345bc4262a1cfe74f53e08c79e6b79960a591014203e9c6e3f50cc5"}
Oct 30 22:52:48.401692 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:48.401616    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n" event={"ID":"c065085c-3963-45a0-b4bc-09ad435e27b7","Type":"ContainerStarted","Data":"cf650300e6dcc0a5ec750cfcc6505bc922bf486e44ee7adeea7cc726f7059b83"}
Oct 30 22:52:48.401692 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:48.401661    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"
Oct 30 22:52:48.517032 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:48.516976    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r" podStartSLOduration=2.516960628 podStartE2EDuration="2.516960628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:48.514505693 +0000 UTC m=+4219.248868724" watchObservedRunningTime="2026-10-30 22:52:48.516960628 +0000 UTC m=+4219.251323659"
Oct 30 22:52:48.632610 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:48.632555    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n" podStartSLOduration=2.632538568 podStartE2EDuration="2.632538568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:48.630376295 +0000 UTC m=+4219.364739325" watchObservedRunningTime="2026-10-30 22:52:48.632538568 +0000 UTC m=+4219.366901598"
Oct 30 22:52:49.115943 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.115856    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"
Oct 30 22:52:49.208869 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.208834    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932-kube-api-access-98hs7\" (UniqueName: \"kubernetes.io/projected/7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932-kube-api-access-98hs7\") pod \"7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932\" (UID: \"7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932\") "
Oct 30 22:52:49.209066 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.208956    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932-plugins-config-volume\") pod \"7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932\" (UID: \"7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932\") "
Oct 30 22:52:49.209382 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.209354    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932-plugins-config-volume" pod "7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932" (UID: "7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:49.211394 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.211365    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932-kube-api-access-98hs7" pod "7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932" (UID: "7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932"). InnerVolumeSpecName "kube-api-access-98hs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:49.310398 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.310360    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98hs7\" (UniqueName: \"kubernetes.io/projected/7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932-kube-api-access-98hs7\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:49.310398 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.310395    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:49.408629 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.408596    4553 generic.go:356] "Generic (PLEG): container finished" podID="7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932" containerID="55a2e1bd9d555415579910511eb9f17bc3dd425cd76995fa1ec9fd911c6eb7ac" exitCode=0
Oct 30 22:52:49.409098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.408643    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"
Oct 30 22:52:49.409098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.408677    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq" event={"ID":"7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932","Type":"ContainerDied","Data":"55a2e1bd9d555415579910511eb9f17bc3dd425cd76995fa1ec9fd911c6eb7ac"}
Oct 30 22:52:49.409098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.408747    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq" event={"ID":"7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932","Type":"ContainerDied","Data":"ecb9261a435dcb2857624b261fa61bdc22320e7f2b3048c191512b5517dc719c"}
Oct 30 22:52:49.409098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.408782    4553 scope.go:122] "RemoveContainer" containerID="55a2e1bd9d555415579910511eb9f17bc3dd425cd76995fa1ec9fd911c6eb7ac"
Oct 30 22:52:49.419924 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.419894    4553 scope.go:122] "RemoveContainer" containerID="55a2e1bd9d555415579910511eb9f17bc3dd425cd76995fa1ec9fd911c6eb7ac"
Oct 30 22:52:49.420261 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:52:49.420239    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a2e1bd9d555415579910511eb9f17bc3dd425cd76995fa1ec9fd911c6eb7ac\": container with ID starting with 55a2e1bd9d555415579910511eb9f17bc3dd425cd76995fa1ec9fd911c6eb7ac not found: ID does not exist" containerID="55a2e1bd9d555415579910511eb9f17bc3dd425cd76995fa1ec9fd911c6eb7ac"
Oct 30 22:52:49.420351 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.420272    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a2e1bd9d555415579910511eb9f17bc3dd425cd76995fa1ec9fd911c6eb7ac"} err="failed to get container status \"55a2e1bd9d555415579910511eb9f17bc3dd425cd76995fa1ec9fd911c6eb7ac\": rpc error: code = NotFound desc = could not find container \"55a2e1bd9d555415579910511eb9f17bc3dd425cd76995fa1ec9fd911c6eb7ac\": container with ID starting with 55a2e1bd9d555415579910511eb9f17bc3dd425cd76995fa1ec9fd911c6eb7ac not found: ID does not exist"
Oct 30 22:52:49.575405 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.575367    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"]
Oct 30 22:52:49.579409 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.579373    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-6c788d-5cb67c5479-fb8gq"]
Oct 30 22:52:49.741358 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.741254    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8" podUID="3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:52:49.741358 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.741261    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq" podUID="289e89bb-9bee-47f8-8768-8ac8c35b47e5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:52:49.987841 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:49.987805    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"]
Oct 30 22:52:50.103857 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:50.103820    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"]
Oct 30 22:52:51.394647 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.394614    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932" path="/var/lib/kubelet/pods/7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932/volumes"
Oct 30 22:52:51.426808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.426779    4553 generic.go:356] "Generic (PLEG): container finished" podID="3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3" containerID="4a87b50dc81c246b1b79f4949bf9e0662c49dcc88222ca114f12469c574aaccd" exitCode=0
Oct 30 22:52:51.426977 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.426860    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8" event={"ID":"3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3","Type":"ContainerDied","Data":"4a87b50dc81c246b1b79f4949bf9e0662c49dcc88222ca114f12469c574aaccd"}
Oct 30 22:52:51.517569 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.517497    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"
Oct 30 22:52:51.633463 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.633070    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"
Oct 30 22:52:51.636318 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.636287    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3-kube-api-access-qz269\" (UniqueName: \"kubernetes.io/projected/3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3-kube-api-access-qz269\") pod \"3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3\" (UID: \"3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3\") "
Oct 30 22:52:51.636457 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.636444    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3-plugins-config-volume\") pod \"3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3\" (UID: \"3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3\") "
Oct 30 22:52:51.636996 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.636934    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3-plugins-config-volume" pod "3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3" (UID: "3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:51.638694 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.638669    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3-kube-api-access-qz269" pod "3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3" (UID: "3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3"). InnerVolumeSpecName "kube-api-access-qz269". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:51.738014 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.737978    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289e89bb-9bee-47f8-8768-8ac8c35b47e5-kube-api-access-j9mbv\" (UniqueName: \"kubernetes.io/projected/289e89bb-9bee-47f8-8768-8ac8c35b47e5-kube-api-access-j9mbv\") pod \"289e89bb-9bee-47f8-8768-8ac8c35b47e5\" (UID: \"289e89bb-9bee-47f8-8768-8ac8c35b47e5\") "
Oct 30 22:52:51.738279 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.738099    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289e89bb-9bee-47f8-8768-8ac8c35b47e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289e89bb-9bee-47f8-8768-8ac8c35b47e5-plugins-config-volume\") pod \"289e89bb-9bee-47f8-8768-8ac8c35b47e5\" (UID: \"289e89bb-9bee-47f8-8768-8ac8c35b47e5\") "
Oct 30 22:52:51.738515 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.738491    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289e89bb-9bee-47f8-8768-8ac8c35b47e5-plugins-config-volume" pod "289e89bb-9bee-47f8-8768-8ac8c35b47e5" (UID: "289e89bb-9bee-47f8-8768-8ac8c35b47e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:51.738515 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.738509    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:51.738636 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.738532    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz269\" (UniqueName: \"kubernetes.io/projected/3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3-kube-api-access-qz269\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:51.740389 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.740368    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289e89bb-9bee-47f8-8768-8ac8c35b47e5-kube-api-access-j9mbv" pod "289e89bb-9bee-47f8-8768-8ac8c35b47e5" (UID: "289e89bb-9bee-47f8-8768-8ac8c35b47e5"). InnerVolumeSpecName "kube-api-access-j9mbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:51.839651 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.839616    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9mbv\" (UniqueName: \"kubernetes.io/projected/289e89bb-9bee-47f8-8768-8ac8c35b47e5-kube-api-access-j9mbv\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:51.839651 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:51.839650    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289e89bb-9bee-47f8-8768-8ac8c35b47e5-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:52.435456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:52.435419    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"
Oct 30 22:52:52.435456 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:52.435440    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8" event={"ID":"3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3","Type":"ContainerDied","Data":"8ad18c0f42a701e384519017d7654a665de1e135848c00486d86673766d4d8c9"}
Oct 30 22:52:52.435976 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:52.435504    4553 scope.go:122] "RemoveContainer" containerID="4a87b50dc81c246b1b79f4949bf9e0662c49dcc88222ca114f12469c574aaccd"
Oct 30 22:52:52.437045 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:52.437009    4553 generic.go:356] "Generic (PLEG): container finished" podID="289e89bb-9bee-47f8-8768-8ac8c35b47e5" containerID="def8e973ef186f5abf22374f8931585193301a7fd01af3b6656cfc40fae9804d" exitCode=0
Oct 30 22:52:52.437215 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:52.437081    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"
Oct 30 22:52:52.437266 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:52.437089    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq" event={"ID":"289e89bb-9bee-47f8-8768-8ac8c35b47e5","Type":"ContainerDied","Data":"def8e973ef186f5abf22374f8931585193301a7fd01af3b6656cfc40fae9804d"}
Oct 30 22:52:52.437310 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:52.437284    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq" event={"ID":"289e89bb-9bee-47f8-8768-8ac8c35b47e5","Type":"ContainerDied","Data":"c563910daea6708c98b86da338ea9d85bc087439d781d23a36eb6a0dd3b7c285"}
Oct 30 22:52:52.447646 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:52.447623    4553 scope.go:122] "RemoveContainer" containerID="def8e973ef186f5abf22374f8931585193301a7fd01af3b6656cfc40fae9804d"
Oct 30 22:52:52.456826 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:52.456806    4553 scope.go:122] "RemoveContainer" containerID="def8e973ef186f5abf22374f8931585193301a7fd01af3b6656cfc40fae9804d"
Oct 30 22:52:52.457110 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:52:52.457078    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"def8e973ef186f5abf22374f8931585193301a7fd01af3b6656cfc40fae9804d\": container with ID starting with def8e973ef186f5abf22374f8931585193301a7fd01af3b6656cfc40fae9804d not found: ID does not exist" containerID="def8e973ef186f5abf22374f8931585193301a7fd01af3b6656cfc40fae9804d"
Oct 30 22:52:52.457239 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:52.457117    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"def8e973ef186f5abf22374f8931585193301a7fd01af3b6656cfc40fae9804d"} err="failed to get container status \"def8e973ef186f5abf22374f8931585193301a7fd01af3b6656cfc40fae9804d\": rpc error: code = NotFound desc = could not find container \"def8e973ef186f5abf22374f8931585193301a7fd01af3b6656cfc40fae9804d\": container with ID starting with def8e973ef186f5abf22374f8931585193301a7fd01af3b6656cfc40fae9804d not found: ID does not exist"
Oct 30 22:52:52.602860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:52.602822    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"]
Oct 30 22:52:52.653192 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:52.653120    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-b1eb3f-58cbfc6c85-f5kb8"]
Oct 30 22:52:52.767200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:52.767162    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"]
Oct 30 22:52:52.776672 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:52.776637    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-b1eb3f-5996948b67-fx5zq"]
Oct 30 22:52:53.394066 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:53.394031    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289e89bb-9bee-47f8-8768-8ac8c35b47e5" path="/var/lib/kubelet/pods/289e89bb-9bee-47f8-8768-8ac8c35b47e5/volumes"
Oct 30 22:52:53.394500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:53.394485    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3" path="/var/lib/kubelet/pods/3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3/volumes"
Oct 30 22:52:54.989040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:54.988994    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72" podUID="d158a3d2-ee91-4ff9-8def-629768a484d2" containerName="payload-processing" containerID="cri-o://ad8dc129821cf2f608f82ca129dc467521d04e64f1b57ca493b2ccd1e80dd8c0" gracePeriod=25
Oct 30 22:52:55.104247 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.104192    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w" podUID="8f0e9ade-f4a6-4795-bda4-63df0eedf974" containerName="payload-pre-processing" containerID="cri-o://46333c4dbc3a2841dab5ad6398483d4c93ec962b7f83979d007ec7f765b8027e" gracePeriod=25
Oct 30 22:52:55.253636 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.253555    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"
Oct 30 22:52:55.367019 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.366994    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"
Oct 30 22:52:55.382700 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.382665    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d158a3d2-ee91-4ff9-8def-629768a484d2-kube-api-access-7kjn8\" (UniqueName: \"kubernetes.io/projected/d158a3d2-ee91-4ff9-8def-629768a484d2-kube-api-access-7kjn8\") pod \"d158a3d2-ee91-4ff9-8def-629768a484d2\" (UID: \"d158a3d2-ee91-4ff9-8def-629768a484d2\") "
Oct 30 22:52:55.382879 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.382827    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d158a3d2-ee91-4ff9-8def-629768a484d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d158a3d2-ee91-4ff9-8def-629768a484d2-plugins-config-volume\") pod \"d158a3d2-ee91-4ff9-8def-629768a484d2\" (UID: \"d158a3d2-ee91-4ff9-8def-629768a484d2\") "
Oct 30 22:52:55.383206 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.383183    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d158a3d2-ee91-4ff9-8def-629768a484d2-plugins-config-volume" pod "d158a3d2-ee91-4ff9-8def-629768a484d2" (UID: "d158a3d2-ee91-4ff9-8def-629768a484d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:55.383357 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.383342    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d158a3d2-ee91-4ff9-8def-629768a484d2-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:55.384935 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.384911    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d158a3d2-ee91-4ff9-8def-629768a484d2-kube-api-access-7kjn8" pod "d158a3d2-ee91-4ff9-8def-629768a484d2" (UID: "d158a3d2-ee91-4ff9-8def-629768a484d2"). InnerVolumeSpecName "kube-api-access-7kjn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:55.458401 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.458368    4553 generic.go:356] "Generic (PLEG): container finished" podID="8f0e9ade-f4a6-4795-bda4-63df0eedf974" containerID="46333c4dbc3a2841dab5ad6398483d4c93ec962b7f83979d007ec7f765b8027e" exitCode=0
Oct 30 22:52:55.458606 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.458420    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"
Oct 30 22:52:55.458606 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.458451    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w" event={"ID":"8f0e9ade-f4a6-4795-bda4-63df0eedf974","Type":"ContainerDied","Data":"46333c4dbc3a2841dab5ad6398483d4c93ec962b7f83979d007ec7f765b8027e"}
Oct 30 22:52:55.458606 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.458504    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w" event={"ID":"8f0e9ade-f4a6-4795-bda4-63df0eedf974","Type":"ContainerDied","Data":"44de5bf11ad2589d64a13c4a5a7f874c19561f04b1d991dee325f18ab89345ce"}
Oct 30 22:52:55.458606 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.458525    4553 scope.go:122] "RemoveContainer" containerID="46333c4dbc3a2841dab5ad6398483d4c93ec962b7f83979d007ec7f765b8027e"
Oct 30 22:52:55.460248 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.460163    4553 generic.go:356] "Generic (PLEG): container finished" podID="d158a3d2-ee91-4ff9-8def-629768a484d2" containerID="ad8dc129821cf2f608f82ca129dc467521d04e64f1b57ca493b2ccd1e80dd8c0" exitCode=0
Oct 30 22:52:55.460248 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.460181    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"
Oct 30 22:52:55.460248 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.460250    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72" event={"ID":"d158a3d2-ee91-4ff9-8def-629768a484d2","Type":"ContainerDied","Data":"ad8dc129821cf2f608f82ca129dc467521d04e64f1b57ca493b2ccd1e80dd8c0"}
Oct 30 22:52:55.460542 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.460290    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72" event={"ID":"d158a3d2-ee91-4ff9-8def-629768a484d2","Type":"ContainerDied","Data":"d75c26a22e48d0eae354bc9673b957eab159ff2e134a9784b6bafd3370450f4a"}
Oct 30 22:52:55.472351 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.472296    4553 scope.go:122] "RemoveContainer" containerID="46333c4dbc3a2841dab5ad6398483d4c93ec962b7f83979d007ec7f765b8027e"
Oct 30 22:52:55.472793 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:52:55.472771    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46333c4dbc3a2841dab5ad6398483d4c93ec962b7f83979d007ec7f765b8027e\": container with ID starting with 46333c4dbc3a2841dab5ad6398483d4c93ec962b7f83979d007ec7f765b8027e not found: ID does not exist" containerID="46333c4dbc3a2841dab5ad6398483d4c93ec962b7f83979d007ec7f765b8027e"
Oct 30 22:52:55.472890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.472804    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46333c4dbc3a2841dab5ad6398483d4c93ec962b7f83979d007ec7f765b8027e"} err="failed to get container status \"46333c4dbc3a2841dab5ad6398483d4c93ec962b7f83979d007ec7f765b8027e\": rpc error: code = NotFound desc = could not find container \"46333c4dbc3a2841dab5ad6398483d4c93ec962b7f83979d007ec7f765b8027e\": container with ID starting with 46333c4dbc3a2841dab5ad6398483d4c93ec962b7f83979d007ec7f765b8027e not found: ID does not exist"
Oct 30 22:52:55.472890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.472825    4553 scope.go:122] "RemoveContainer" containerID="ad8dc129821cf2f608f82ca129dc467521d04e64f1b57ca493b2ccd1e80dd8c0"
Oct 30 22:52:55.482278 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.482256    4553 scope.go:122] "RemoveContainer" containerID="ad8dc129821cf2f608f82ca129dc467521d04e64f1b57ca493b2ccd1e80dd8c0"
Oct 30 22:52:55.482560 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:52:55.482540    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad8dc129821cf2f608f82ca129dc467521d04e64f1b57ca493b2ccd1e80dd8c0\": container with ID starting with ad8dc129821cf2f608f82ca129dc467521d04e64f1b57ca493b2ccd1e80dd8c0 not found: ID does not exist" containerID="ad8dc129821cf2f608f82ca129dc467521d04e64f1b57ca493b2ccd1e80dd8c0"
Oct 30 22:52:55.482602 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.482568    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad8dc129821cf2f608f82ca129dc467521d04e64f1b57ca493b2ccd1e80dd8c0"} err="failed to get container status \"ad8dc129821cf2f608f82ca129dc467521d04e64f1b57ca493b2ccd1e80dd8c0\": rpc error: code = NotFound desc = could not find container \"ad8dc129821cf2f608f82ca129dc467521d04e64f1b57ca493b2ccd1e80dd8c0\": container with ID starting with ad8dc129821cf2f608f82ca129dc467521d04e64f1b57ca493b2ccd1e80dd8c0 not found: ID does not exist"
Oct 30 22:52:55.483913 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.483896    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0e9ade-f4a6-4795-bda4-63df0eedf974-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0e9ade-f4a6-4795-bda4-63df0eedf974-plugins-config-volume\") pod \"8f0e9ade-f4a6-4795-bda4-63df0eedf974\" (UID: \"8f0e9ade-f4a6-4795-bda4-63df0eedf974\") "
Oct 30 22:52:55.484026 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.484007    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0e9ade-f4a6-4795-bda4-63df0eedf974-kube-api-access-4dtx6\" (UniqueName: \"kubernetes.io/projected/8f0e9ade-f4a6-4795-bda4-63df0eedf974-kube-api-access-4dtx6\") pod \"8f0e9ade-f4a6-4795-bda4-63df0eedf974\" (UID: \"8f0e9ade-f4a6-4795-bda4-63df0eedf974\") "
Oct 30 22:52:55.484335 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.484320    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kjn8\" (UniqueName: \"kubernetes.io/projected/d158a3d2-ee91-4ff9-8def-629768a484d2-kube-api-access-7kjn8\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:55.484518 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.484497    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0e9ade-f4a6-4795-bda4-63df0eedf974-plugins-config-volume" pod "8f0e9ade-f4a6-4795-bda4-63df0eedf974" (UID: "8f0e9ade-f4a6-4795-bda4-63df0eedf974"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:55.486188 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.486164    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0e9ade-f4a6-4795-bda4-63df0eedf974-kube-api-access-4dtx6" pod "8f0e9ade-f4a6-4795-bda4-63df0eedf974" (UID: "8f0e9ade-f4a6-4795-bda4-63df0eedf974"). InnerVolumeSpecName "kube-api-access-4dtx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:55.585591 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.585558    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0e9ade-f4a6-4795-bda4-63df0eedf974-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:55.585591 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.585591    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dtx6\" (UniqueName: \"kubernetes.io/projected/8f0e9ade-f4a6-4795-bda4-63df0eedf974-kube-api-access-4dtx6\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:55.660685 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.660653    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"]
Oct 30 22:52:55.664246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.664221    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b8717e-66555bd5dc-dbp72"]
Oct 30 22:52:55.882553 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.882471    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"]
Oct 30 22:52:55.884991 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:55.884963    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b8717e-545855f4c9-d792w"]
Oct 30 22:52:57.401391 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:57.401355    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0e9ade-f4a6-4795-bda4-63df0eedf974" path="/var/lib/kubelet/pods/8f0e9ade-f4a6-4795-bda4-63df0eedf974/volumes"
Oct 30 22:52:57.401813 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:57.401764    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d158a3d2-ee91-4ff9-8def-629768a484d2" path="/var/lib/kubelet/pods/d158a3d2-ee91-4ff9-8def-629768a484d2/volumes"
Oct 30 22:52:58.202106 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:52:58.202071    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:52:58.427908 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.427865    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main" containerID="cri-o://7a8aeba4946931c896039d8dbc2c76f14d4462f350e74771cd2371f94460016a" gracePeriod=45
Oct 30 22:52:58.478009 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.477920    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" containerID="cri-o://d7df96d81c4953fb894ea174428339fd378306fae3c9c5ddc44d77b04f73ee8c" gracePeriod=45
Oct 30 22:52:58.737118 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.737043    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:52:58.775471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.775443    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 22:52:58.821390 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.821343    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f2de714-796f-4938-8325-0004f2c22a02-kube-api-access-628kx\" (UniqueName: \"kubernetes.io/projected/1f2de714-796f-4938-8325-0004f2c22a02-kube-api-access-628kx\") pod \"1f2de714-796f-4938-8325-0004f2c22a02\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") "
Oct 30 22:52:58.821611 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.821402    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-tmp-dir\") pod \"1f2de714-796f-4938-8325-0004f2c22a02\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") "
Oct 30 22:52:58.821611 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.821475    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-home\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-home\") pod \"1f2de714-796f-4938-8325-0004f2c22a02\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") "
Oct 30 22:52:58.821611 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.821543    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f2de714-796f-4938-8325-0004f2c22a02-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f2de714-796f-4938-8325-0004f2c22a02-tls-certs\") pod \"1f2de714-796f-4938-8325-0004f2c22a02\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") "
Oct 30 22:52:58.821611 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.821587    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-dshm\") pod \"1f2de714-796f-4938-8325-0004f2c22a02\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") "
Oct 30 22:52:58.821859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.821684    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-tmp-dir" pod "1f2de714-796f-4938-8325-0004f2c22a02" (UID: "1f2de714-796f-4938-8325-0004f2c22a02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:58.821859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.821725    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-model-cache\") pod \"1f2de714-796f-4938-8325-0004f2c22a02\" (UID: \"1f2de714-796f-4938-8325-0004f2c22a02\") "
Oct 30 22:52:58.821979 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.821876    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-home" pod "1f2de714-796f-4938-8325-0004f2c22a02" (UID: "1f2de714-796f-4938-8325-0004f2c22a02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:58.822092 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.822070    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-model-cache" pod "1f2de714-796f-4938-8325-0004f2c22a02" (UID: "1f2de714-796f-4938-8325-0004f2c22a02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:58.822438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.822301    4553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-tmp-dir\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:58.822438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.822323    4553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-home\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:58.822438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.822341    4553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-model-cache\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:58.824576 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.824537    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-dshm" pod "1f2de714-796f-4938-8325-0004f2c22a02" (UID: "1f2de714-796f-4938-8325-0004f2c22a02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:58.826003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.825964    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f2de714-796f-4938-8325-0004f2c22a02-tls-certs" pod "1f2de714-796f-4938-8325-0004f2c22a02" (UID: "1f2de714-796f-4938-8325-0004f2c22a02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:58.826121 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.826086    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2de714-796f-4938-8325-0004f2c22a02-kube-api-access-628kx" pod "1f2de714-796f-4938-8325-0004f2c22a02" (UID: "1f2de714-796f-4938-8325-0004f2c22a02"). InnerVolumeSpecName "kube-api-access-628kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:58.922946 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.922914    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7a0ddb-718a-41f3-8fbd-e83896d80587-kube-api-access-lj8px\" (UniqueName: \"kubernetes.io/projected/fd7a0ddb-718a-41f3-8fbd-e83896d80587-kube-api-access-lj8px\") pod \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") "
Oct 30 22:52:58.923255 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.922974    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd7a0ddb-718a-41f3-8fbd-e83896d80587-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd7a0ddb-718a-41f3-8fbd-e83896d80587-tls-certs\") pod \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") "
Oct 30 22:52:58.923255 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.923006    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-dshm\") pod \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") "
Oct 30 22:52:58.923255 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.923216    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-tmp-dir\") pod \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") "
Oct 30 22:52:58.923484 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.923278    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-home\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-home\") pod \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") "
Oct 30 22:52:58.923484 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.923322    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-model-cache\") pod \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\" (UID: \"fd7a0ddb-718a-41f3-8fbd-e83896d80587\") "
Oct 30 22:52:58.923603 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.923489    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-tmp-dir" pod "fd7a0ddb-718a-41f3-8fbd-e83896d80587" (UID: "fd7a0ddb-718a-41f3-8fbd-e83896d80587"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:58.923603 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.923544    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-home" pod "fd7a0ddb-718a-41f3-8fbd-e83896d80587" (UID: "fd7a0ddb-718a-41f3-8fbd-e83896d80587"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:58.923787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.923738    4553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f2de714-796f-4938-8325-0004f2c22a02-tls-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:58.923787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.923753    4553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-tmp-dir\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:58.923787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.923766    4553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f2de714-796f-4938-8325-0004f2c22a02-dshm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:58.923787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.923777    4553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-home\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:58.923787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.923791    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-628kx\" (UniqueName: \"kubernetes.io/projected/1f2de714-796f-4938-8325-0004f2c22a02-kube-api-access-628kx\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:58.924179 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.923830    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-model-cache" pod "fd7a0ddb-718a-41f3-8fbd-e83896d80587" (UID: "fd7a0ddb-718a-41f3-8fbd-e83896d80587"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:58.925246 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.925220    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd7a0ddb-718a-41f3-8fbd-e83896d80587-tls-certs" pod "fd7a0ddb-718a-41f3-8fbd-e83896d80587" (UID: "fd7a0ddb-718a-41f3-8fbd-e83896d80587"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:58.925353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.925259    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7a0ddb-718a-41f3-8fbd-e83896d80587-kube-api-access-lj8px" pod "fd7a0ddb-718a-41f3-8fbd-e83896d80587" (UID: "fd7a0ddb-718a-41f3-8fbd-e83896d80587"). InnerVolumeSpecName "kube-api-access-lj8px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:58.925353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:58.925323    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-dshm" pod "fd7a0ddb-718a-41f3-8fbd-e83896d80587" (UID: "fd7a0ddb-718a-41f3-8fbd-e83896d80587"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:59.025280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.025205    4553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-model-cache\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:59.025280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.025238    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj8px\" (UniqueName: \"kubernetes.io/projected/fd7a0ddb-718a-41f3-8fbd-e83896d80587-kube-api-access-lj8px\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:59.025280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.025255    4553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd7a0ddb-718a-41f3-8fbd-e83896d80587-tls-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:59.025280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.025265    4553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd7a0ddb-718a-41f3-8fbd-e83896d80587-dshm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:52:59.412334 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.412310    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"
Oct 30 22:52:59.412501 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.412388    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"
Oct 30 22:52:59.485190 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.485155    4553 generic.go:356] "Generic (PLEG): container finished" podID="1f2de714-796f-4938-8325-0004f2c22a02" containerID="7a8aeba4946931c896039d8dbc2c76f14d4462f350e74771cd2371f94460016a" exitCode=0
Oct 30 22:52:59.485731 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.485200    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"
Oct 30 22:52:59.485731 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.485229    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" event={"ID":"1f2de714-796f-4938-8325-0004f2c22a02","Type":"ContainerDied","Data":"7a8aeba4946931c896039d8dbc2c76f14d4462f350e74771cd2371f94460016a"}
Oct 30 22:52:59.485731 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.485286    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc" event={"ID":"1f2de714-796f-4938-8325-0004f2c22a02","Type":"ContainerDied","Data":"46414c076b9beb17adecdf54f3135e4d71777ea218a871cda64b716c81dc8b5c"}
Oct 30 22:52:59.485731 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.485310    4553 scope.go:122] "RemoveContainer" containerID="7a8aeba4946931c896039d8dbc2c76f14d4462f350e74771cd2371f94460016a"
Oct 30 22:52:59.486960 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.486924    4553 generic.go:356] "Generic (PLEG): container finished" podID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerID="d7df96d81c4953fb894ea174428339fd378306fae3c9c5ddc44d77b04f73ee8c" exitCode=0
Oct 30 22:52:59.487077 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.486975    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"
Oct 30 22:52:59.487077 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.486975    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" event={"ID":"fd7a0ddb-718a-41f3-8fbd-e83896d80587","Type":"ContainerDied","Data":"d7df96d81c4953fb894ea174428339fd378306fae3c9c5ddc44d77b04f73ee8c"}
Oct 30 22:52:59.487077 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.487014    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" event={"ID":"fd7a0ddb-718a-41f3-8fbd-e83896d80587","Type":"ContainerDied","Data":"60479501c262f3de93940d9f403797833a770d89163094ed11fd9bcbb13f486d"}
Oct 30 22:52:59.496782 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.496753    4553 scope.go:122] "RemoveContainer" containerID="7a8aeba4946931c896039d8dbc2c76f14d4462f350e74771cd2371f94460016a"
Oct 30 22:52:59.497187 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:52:59.497161    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8aeba4946931c896039d8dbc2c76f14d4462f350e74771cd2371f94460016a\": container with ID starting with 7a8aeba4946931c896039d8dbc2c76f14d4462f350e74771cd2371f94460016a not found: ID does not exist" containerID="7a8aeba4946931c896039d8dbc2c76f14d4462f350e74771cd2371f94460016a"
Oct 30 22:52:59.497279 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.497201    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8aeba4946931c896039d8dbc2c76f14d4462f350e74771cd2371f94460016a"} err="failed to get container status \"7a8aeba4946931c896039d8dbc2c76f14d4462f350e74771cd2371f94460016a\": rpc error: code = NotFound desc = could not find container \"7a8aeba4946931c896039d8dbc2c76f14d4462f350e74771cd2371f94460016a\": container with ID starting with 7a8aeba4946931c896039d8dbc2c76f14d4462f350e74771cd2371f94460016a not found: ID does not exist"
Oct 30 22:52:59.497279 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.497228    4553 scope.go:122] "RemoveContainer" containerID="d7df96d81c4953fb894ea174428339fd378306fae3c9c5ddc44d77b04f73ee8c"
Oct 30 22:52:59.506282 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.506261    4553 scope.go:122] "RemoveContainer" containerID="d7df96d81c4953fb894ea174428339fd378306fae3c9c5ddc44d77b04f73ee8c"
Oct 30 22:52:59.506576 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:52:59.506557    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7df96d81c4953fb894ea174428339fd378306fae3c9c5ddc44d77b04f73ee8c\": container with ID starting with d7df96d81c4953fb894ea174428339fd378306fae3c9c5ddc44d77b04f73ee8c not found: ID does not exist" containerID="d7df96d81c4953fb894ea174428339fd378306fae3c9c5ddc44d77b04f73ee8c"
Oct 30 22:52:59.506645 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.506585    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7df96d81c4953fb894ea174428339fd378306fae3c9c5ddc44d77b04f73ee8c"} err="failed to get container status \"d7df96d81c4953fb894ea174428339fd378306fae3c9c5ddc44d77b04f73ee8c\": rpc error: code = NotFound desc = could not find container \"d7df96d81c4953fb894ea174428339fd378306fae3c9c5ddc44d77b04f73ee8c\": container with ID starting with d7df96d81c4953fb894ea174428339fd378306fae3c9c5ddc44d77b04f73ee8c not found: ID does not exist"
Oct 30 22:52:59.754853 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.754754    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"]
Oct 30 22:52:59.758886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.758846    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/distinct-w0-b8717e-kserve-544fffc6bc-kj7wc"]
Oct 30 22:52:59.765533 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.765478    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:52:59.874262 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.874218    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"]
Oct 30 22:52:59.878880 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:52:59.878847    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b8717e/premium-w0-b8717e-kserve-dd67f47b8-kmqr4"]
Oct 30 22:53:01.391993 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:01.391962    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f2de714-796f-4938-8325-0004f2c22a02" path="/var/lib/kubelet/pods/1f2de714-796f-4938-8325-0004f2c22a02/volumes"
Oct 30 22:53:01.392411 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:01.392396    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" path="/var/lib/kubelet/pods/fd7a0ddb-718a-41f3-8fbd-e83896d80587/volumes"
Oct 30 22:53:04.541089 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:04.541051    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"]
Oct 30 22:53:04.541563 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:04.541273    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6" podUID="944271c0-d01e-4e9c-9c48-a0b892421f5b" containerName="maas-api" containerID="cri-o://f19b43190683653f8adc91b2ca3916e64c8e54e766f2b34857831499f1cc9d1c" gracePeriod=30
Oct 30 22:53:04.795883 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:04.795822    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:53:04.886595 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:04.886561    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-maas-api-tls\") pod \"944271c0-d01e-4e9c-9c48-a0b892421f5b\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") "
Oct 30 22:53:04.886793 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:04.886637    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944271c0-d01e-4e9c-9c48-a0b892421f5b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/944271c0-d01e-4e9c-9c48-a0b892421f5b-openshift-service-ca\") pod \"944271c0-d01e-4e9c-9c48-a0b892421f5b\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") "
Oct 30 22:53:04.886793 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:04.886690    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944271c0-d01e-4e9c-9c48-a0b892421f5b-kube-api-access-wwqwd\" (UniqueName: \"kubernetes.io/projected/944271c0-d01e-4e9c-9c48-a0b892421f5b-kube-api-access-wwqwd\") pod \"944271c0-d01e-4e9c-9c48-a0b892421f5b\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") "
Oct 30 22:53:04.886876 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:04.886845    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-metrics-certs\") pod \"944271c0-d01e-4e9c-9c48-a0b892421f5b\" (UID: \"944271c0-d01e-4e9c-9c48-a0b892421f5b\") "
Oct 30 22:53:04.887007 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:04.886975    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944271c0-d01e-4e9c-9c48-a0b892421f5b-openshift-service-ca" pod "944271c0-d01e-4e9c-9c48-a0b892421f5b" (UID: "944271c0-d01e-4e9c-9c48-a0b892421f5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:04.887359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:04.887309    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/944271c0-d01e-4e9c-9c48-a0b892421f5b-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:04.888978 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:04.888956    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-maas-api-tls" pod "944271c0-d01e-4e9c-9c48-a0b892421f5b" (UID: "944271c0-d01e-4e9c-9c48-a0b892421f5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:04.889088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:04.889003    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-metrics-certs" pod "944271c0-d01e-4e9c-9c48-a0b892421f5b" (UID: "944271c0-d01e-4e9c-9c48-a0b892421f5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:04.889088 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:04.889038    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944271c0-d01e-4e9c-9c48-a0b892421f5b-kube-api-access-wwqwd" pod "944271c0-d01e-4e9c-9c48-a0b892421f5b" (UID: "944271c0-d01e-4e9c-9c48-a0b892421f5b"). InnerVolumeSpecName "kube-api-access-wwqwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:04.988424 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:04.988378    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:04.988424 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:04.988411    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/944271c0-d01e-4e9c-9c48-a0b892421f5b-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:04.988424 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:04.988421    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwqwd\" (UniqueName: \"kubernetes.io/projected/944271c0-d01e-4e9c-9c48-a0b892421f5b-kube-api-access-wwqwd\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:05.522549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:05.522511    4553 generic.go:356] "Generic (PLEG): container finished" podID="944271c0-d01e-4e9c-9c48-a0b892421f5b" containerID="f19b43190683653f8adc91b2ca3916e64c8e54e766f2b34857831499f1cc9d1c" exitCode=0
Oct 30 22:53:05.522742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:05.522554    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"
Oct 30 22:53:05.522742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:05.522599    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6" event={"ID":"944271c0-d01e-4e9c-9c48-a0b892421f5b","Type":"ContainerDied","Data":"f19b43190683653f8adc91b2ca3916e64c8e54e766f2b34857831499f1cc9d1c"}
Oct 30 22:53:05.522742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:05.522656    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6" event={"ID":"944271c0-d01e-4e9c-9c48-a0b892421f5b","Type":"ContainerDied","Data":"f6c482071c6cba5373fb726eb522c01f1c7e3518371a9b2e8f417041745fe809"}
Oct 30 22:53:05.522742 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:05.522686    4553 scope.go:122] "RemoveContainer" containerID="f19b43190683653f8adc91b2ca3916e64c8e54e766f2b34857831499f1cc9d1c"
Oct 30 22:53:05.532226 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:05.532200    4553 scope.go:122] "RemoveContainer" containerID="f19b43190683653f8adc91b2ca3916e64c8e54e766f2b34857831499f1cc9d1c"
Oct 30 22:53:05.532509 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:53:05.532487    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19b43190683653f8adc91b2ca3916e64c8e54e766f2b34857831499f1cc9d1c\": container with ID starting with f19b43190683653f8adc91b2ca3916e64c8e54e766f2b34857831499f1cc9d1c not found: ID does not exist" containerID="f19b43190683653f8adc91b2ca3916e64c8e54e766f2b34857831499f1cc9d1c"
Oct 30 22:53:05.532578 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:05.532519    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19b43190683653f8adc91b2ca3916e64c8e54e766f2b34857831499f1cc9d1c"} err="failed to get container status \"f19b43190683653f8adc91b2ca3916e64c8e54e766f2b34857831499f1cc9d1c\": rpc error: code = NotFound desc = could not find container \"f19b43190683653f8adc91b2ca3916e64c8e54e766f2b34857831499f1cc9d1c\": container with ID starting with f19b43190683653f8adc91b2ca3916e64c8e54e766f2b34857831499f1cc9d1c not found: ID does not exist"
Oct 30 22:53:05.686509 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:05.686408    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"]
Oct 30 22:53:05.687001 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:05.686700    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8" podUID="62e0da6e-0907-469d-8d47-ff4eeb9623ac" containerName="maas-api" containerID="cri-o://dc3b64fa5dca2e8931610c916dbf9cb40fc785fcaa75b3f3a75a3247de59ce26" gracePeriod=30
Oct 30 22:53:05.691184 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:05.691155    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"]
Oct 30 22:53:05.738605 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:05.738574    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6c788d-594c476f4-79mn6"]
Oct 30 22:53:05.952732 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:05.952669    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 22:53:06.103016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.102983    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e0da6e-0907-469d-8d47-ff4eeb9623ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e0da6e-0907-469d-8d47-ff4eeb9623ac-openshift-service-ca\") pod \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") "
Oct 30 22:53:06.103239 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.103036    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-metrics-certs\") pod \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") "
Oct 30 22:53:06.103239 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.103222    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-maas-api-tls\") pod \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") "
Oct 30 22:53:06.103370 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.103258    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e0da6e-0907-469d-8d47-ff4eeb9623ac-kube-api-access-vs54d\" (UniqueName: \"kubernetes.io/projected/62e0da6e-0907-469d-8d47-ff4eeb9623ac-kube-api-access-vs54d\") pod \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\" (UID: \"62e0da6e-0907-469d-8d47-ff4eeb9623ac\") "
Oct 30 22:53:06.103457 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.103425    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e0da6e-0907-469d-8d47-ff4eeb9623ac-openshift-service-ca" pod "62e0da6e-0907-469d-8d47-ff4eeb9623ac" (UID: "62e0da6e-0907-469d-8d47-ff4eeb9623ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:06.103629 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.103613    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e0da6e-0907-469d-8d47-ff4eeb9623ac-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:06.105536 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.105506    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-maas-api-tls" pod "62e0da6e-0907-469d-8d47-ff4eeb9623ac" (UID: "62e0da6e-0907-469d-8d47-ff4eeb9623ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:06.106016 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.105997    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e0da6e-0907-469d-8d47-ff4eeb9623ac-kube-api-access-vs54d" pod "62e0da6e-0907-469d-8d47-ff4eeb9623ac" (UID: "62e0da6e-0907-469d-8d47-ff4eeb9623ac"). InnerVolumeSpecName "kube-api-access-vs54d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:06.106100 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.106003    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-metrics-certs" pod "62e0da6e-0907-469d-8d47-ff4eeb9623ac" (UID: "62e0da6e-0907-469d-8d47-ff4eeb9623ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:06.205350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.205262    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:06.205350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.205291    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e0da6e-0907-469d-8d47-ff4eeb9623ac-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:06.205350 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.205305    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs54d\" (UniqueName: \"kubernetes.io/projected/62e0da6e-0907-469d-8d47-ff4eeb9623ac-kube-api-access-vs54d\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:06.530592 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.530505    4553 generic.go:356] "Generic (PLEG): container finished" podID="62e0da6e-0907-469d-8d47-ff4eeb9623ac" containerID="dc3b64fa5dca2e8931610c916dbf9cb40fc785fcaa75b3f3a75a3247de59ce26" exitCode=0
Oct 30 22:53:06.530592 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.530551    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"
Oct 30 22:53:06.530797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.530587    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8" event={"ID":"62e0da6e-0907-469d-8d47-ff4eeb9623ac","Type":"ContainerDied","Data":"dc3b64fa5dca2e8931610c916dbf9cb40fc785fcaa75b3f3a75a3247de59ce26"}
Oct 30 22:53:06.530797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.530643    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8" event={"ID":"62e0da6e-0907-469d-8d47-ff4eeb9623ac","Type":"ContainerDied","Data":"aa7346e18e6389e7b5a9b880f996bc20b6dc3f97cf9926ac74cf6ac0e7331bc6"}
Oct 30 22:53:06.530797 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.530672    4553 scope.go:122] "RemoveContainer" containerID="dc3b64fa5dca2e8931610c916dbf9cb40fc785fcaa75b3f3a75a3247de59ce26"
Oct 30 22:53:06.541395 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.541375    4553 scope.go:122] "RemoveContainer" containerID="dc3b64fa5dca2e8931610c916dbf9cb40fc785fcaa75b3f3a75a3247de59ce26"
Oct 30 22:53:06.541718 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:53:06.541699    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc3b64fa5dca2e8931610c916dbf9cb40fc785fcaa75b3f3a75a3247de59ce26\": container with ID starting with dc3b64fa5dca2e8931610c916dbf9cb40fc785fcaa75b3f3a75a3247de59ce26 not found: ID does not exist" containerID="dc3b64fa5dca2e8931610c916dbf9cb40fc785fcaa75b3f3a75a3247de59ce26"
Oct 30 22:53:06.541762 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.541728    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc3b64fa5dca2e8931610c916dbf9cb40fc785fcaa75b3f3a75a3247de59ce26"} err="failed to get container status \"dc3b64fa5dca2e8931610c916dbf9cb40fc785fcaa75b3f3a75a3247de59ce26\": rpc error: code = NotFound desc = could not find container \"dc3b64fa5dca2e8931610c916dbf9cb40fc785fcaa75b3f3a75a3247de59ce26\": container with ID starting with dc3b64fa5dca2e8931610c916dbf9cb40fc785fcaa75b3f3a75a3247de59ce26 not found: ID does not exist"
Oct 30 22:53:06.697040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.697004    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"]
Oct 30 22:53:06.701542 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:06.701514    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-d2a3e0-7bb4dbc799-bscf8"]
Oct 30 22:53:07.392798 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:07.392761    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e0da6e-0907-469d-8d47-ff4eeb9623ac" path="/var/lib/kubelet/pods/62e0da6e-0907-469d-8d47-ff4eeb9623ac/volumes"
Oct 30 22:53:07.393233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:07.393218    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944271c0-d01e-4e9c-9c48-a0b892421f5b" path="/var/lib/kubelet/pods/944271c0-d01e-4e9c-9c48-a0b892421f5b/volumes"
Oct 30 22:53:18.372323 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372289    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"]
Oct 30 22:53:18.372848 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372820    4553 cpu_manager.go:414] "containerMap: removing container" podUID="3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3" containerName="payload-processing"
Oct 30 22:53:18.372848 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372832    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3" containerName="payload-processing"
Oct 30 22:53:18.372848 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372845    4553 cpu_manager.go:414] "containerMap: removing container" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main"
Oct 30 22:53:18.372848 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372850    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main"
Oct 30 22:53:18.372995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372864    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932" containerName="payload-pre-processing"
Oct 30 22:53:18.372995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372869    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932" containerName="payload-pre-processing"
Oct 30 22:53:18.372995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372879    4553 cpu_manager.go:414] "containerMap: removing container" podUID="289e89bb-9bee-47f8-8768-8ac8c35b47e5" containerName="payload-pre-processing"
Oct 30 22:53:18.372995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372889    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289e89bb-9bee-47f8-8768-8ac8c35b47e5" containerName="payload-pre-processing"
Oct 30 22:53:18.372995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372900    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d158a3d2-ee91-4ff9-8def-629768a484d2" containerName="payload-processing"
Oct 30 22:53:18.372995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372906    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d158a3d2-ee91-4ff9-8def-629768a484d2" containerName="payload-processing"
Oct 30 22:53:18.372995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372913    4553 cpu_manager.go:414] "containerMap: removing container" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main"
Oct 30 22:53:18.372995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372918    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main"
Oct 30 22:53:18.372995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372934    4553 cpu_manager.go:414] "containerMap: removing container" podUID="944271c0-d01e-4e9c-9c48-a0b892421f5b" containerName="maas-api"
Oct 30 22:53:18.372995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372938    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944271c0-d01e-4e9c-9c48-a0b892421f5b" containerName="maas-api"
Oct 30 22:53:18.372995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372945    4553 cpu_manager.go:414] "containerMap: removing container" podUID="8f0e9ade-f4a6-4795-bda4-63df0eedf974" containerName="payload-pre-processing"
Oct 30 22:53:18.372995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372949    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0e9ade-f4a6-4795-bda4-63df0eedf974" containerName="payload-pre-processing"
Oct 30 22:53:18.372995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372970    4553 cpu_manager.go:414] "containerMap: removing container" podUID="62e0da6e-0907-469d-8d47-ff4eeb9623ac" containerName="maas-api"
Oct 30 22:53:18.372995 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.372975    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e0da6e-0907-469d-8d47-ff4eeb9623ac" containerName="maas-api"
Oct 30 22:53:18.373450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.373052    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="944271c0-d01e-4e9c-9c48-a0b892421f5b" containerName="maas-api"
Oct 30 22:53:18.373450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.373061    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0e9ade-f4a6-4795-bda4-63df0eedf974" containerName="payload-pre-processing"
Oct 30 22:53:18.373450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.373070    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e0da6e-0907-469d-8d47-ff4eeb9623ac" containerName="maas-api"
Oct 30 22:53:18.373450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.373078    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8a2f0e-ff56-4593-aad9-6c0a5dd48ac3" containerName="payload-processing"
Oct 30 22:53:18.373450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.373085    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7a0ddb-718a-41f3-8fbd-e83896d80587" containerName="main"
Oct 30 22:53:18.373450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.373091    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d3bd36e-4828-4ea5-ac71-d8f8f1d0b932" containerName="payload-pre-processing"
Oct 30 22:53:18.373450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.373098    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="289e89bb-9bee-47f8-8768-8ac8c35b47e5" containerName="payload-pre-processing"
Oct 30 22:53:18.373450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.373104    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d158a3d2-ee91-4ff9-8def-629768a484d2" containerName="payload-processing"
Oct 30 22:53:18.373450 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.373109    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f2de714-796f-4938-8325-0004f2c22a02" containerName="main"
Oct 30 22:53:18.376642 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.376625    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:18.429682 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.429652    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-3faf84-serving-cert\""
Oct 30 22:53:18.434413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.434379    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbtw5\" (UniqueName: \"kubernetes.io/projected/62d53659-7ffd-4b08-ae37-8e705017b7d9-kube-api-access-pbtw5\") pod \"maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:18.434568 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.434530    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d53659-7ffd-4b08-ae37-8e705017b7d9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:18.434629 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.434562    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:18.434707 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.434679    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-metrics-certs\") pod \"maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:18.484355 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.484315    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"]
Oct 30 22:53:18.536357 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.536320    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbtw5\" (UniqueName: \"kubernetes.io/projected/62d53659-7ffd-4b08-ae37-8e705017b7d9-kube-api-access-pbtw5\") pod \"maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:18.536601 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.536391    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d53659-7ffd-4b08-ae37-8e705017b7d9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:18.536601 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.536409    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:18.536601 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.536441    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-metrics-certs\") pod \"maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:18.536601 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:53:18.536568    4553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-serving-cert: secret "maas-api-e2e-extmod-b-3faf84-serving-cert" not found
Oct 30 22:53:18.536938 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:53:18.536662    4553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-maas-api-tls podName:62d53659-7ffd-4b08-ae37-8e705017b7d9 nodeName:}" failed. No retries permitted until 2026-10-30 22:53:19.036640148 +0000 UTC m=+4249.771003171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-maas-api-tls") pod "maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4" (UID: "62d53659-7ffd-4b08-ae37-8e705017b7d9") : secret "maas-api-e2e-extmod-b-3faf84-serving-cert" not found
Oct 30 22:53:18.537237 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.537211    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d53659-7ffd-4b08-ae37-8e705017b7d9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:18.539308 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.539287    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-metrics-certs\") pod \"maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:18.594882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:18.594854    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbtw5\" (UniqueName: \"kubernetes.io/projected/62d53659-7ffd-4b08-ae37-8e705017b7d9-kube-api-access-pbtw5\") pod \"maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:19.041614 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:19.041570    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:19.044478 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:19.044449    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:19.293113 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:19.293008    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:19.670719 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:19.670682    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"]
Oct 30 22:53:19.675519 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:53:19.675483    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d53659_7ffd_4b08_ae37_8e705017b7d9.slice/crio-74b718ddab37b34c9a1892e7853209b20c3f76162692819af9af6b60479ed7e2 WatchSource:0}: Error finding container 74b718ddab37b34c9a1892e7853209b20c3f76162692819af9af6b60479ed7e2: Status 404 returned error can't find the container with id 74b718ddab37b34c9a1892e7853209b20c3f76162692819af9af6b60479ed7e2
Oct 30 22:53:20.615003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:20.614955    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4" event={"ID":"62d53659-7ffd-4b08-ae37-8e705017b7d9","Type":"ContainerStarted","Data":"74b718ddab37b34c9a1892e7853209b20c3f76162692819af9af6b60479ed7e2"}
Oct 30 22:53:21.621857 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:21.621823    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4" event={"ID":"62d53659-7ffd-4b08-ae37-8e705017b7d9","Type":"ContainerStarted","Data":"ba1f5d5dc8ebabd4a47453508d6b5214e3d8a9c6d4b17647e460e9d058352bb1"}
Oct 30 22:53:21.622302 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:21.621897    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:21.747388 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:21.747334    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4" podStartSLOduration=2.806267531 podStartE2EDuration="3.747317979s" totalImagesPullingTime="941.050448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:21.745300731 +0000 UTC m=+4252.479663762" watchObservedRunningTime="2026-10-30 22:53:21.747317979 +0000 UTC m=+4252.481681041"
Oct 30 22:53:27.634899 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:27.634870    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:53:28.204100 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:53:28.204069    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:53:29.294515 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:29.294471    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"]
Oct 30 22:53:38.283377 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:38.283325    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:53:38.283889 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:38.283401    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh" podUID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:53:40.729333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:40.729243    4553 generic.go:356] "Generic (PLEG): container finished" podID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerID="3e70ceac94de7789efc6e17df2f2d46f661a842be52c3316f7017e9485b0e487" exitCode=0
Oct 30 22:53:40.729333 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:40.729316    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh" event={"ID":"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b","Type":"ContainerDied","Data":"3e70ceac94de7789efc6e17df2f2d46f661a842be52c3316f7017e9485b0e487"}
Oct 30 22:53:40.729850 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:40.729763    4553 scope.go:122] "RemoveContainer" containerID="3e70ceac94de7789efc6e17df2f2d46f661a842be52c3316f7017e9485b0e487"
Oct 30 22:53:40.731365 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:40.731333    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:53:40.731535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:40.731423    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:53:40.731535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:40.731466    4553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:53:41.742779 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:41.742743    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh" event={"ID":"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b","Type":"ContainerStarted","Data":"9ddb0867779410ab41c3cec7040f2092748b02f4d3562f712424257bc7374fea"}
Oct 30 22:53:42.685458 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:42.685418    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:53:42.685739 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:42.685715    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:53:42.685795 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:42.685777    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh" podUID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 22:53:43.685511 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:43.685473    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:53:43.685972 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:43.685542    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh" podUID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 22:53:44.334280 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.334233    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v" podUID="12e366ed-b8f7-4d11-aae0-9f3e4015c8cc" containerName="main" containerID="cri-o://f5db27f72443ee9a4ece8a6991e991d858f5cafae9dfb0234c521875e399cafc" gracePeriod=45
Oct 30 22:53:44.644032 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.643999    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:53:44.665974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.665945    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-home\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-home\") pod \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") "
Oct 30 22:53:44.666158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.666007    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-dshm\") pod \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") "
Oct 30 22:53:44.666158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.666028    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-model-cache\") pod \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") "
Oct 30 22:53:44.666158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.666059    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-kube-api-access-k8rt8\" (UniqueName: \"kubernetes.io/projected/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-kube-api-access-k8rt8\") pod \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") "
Oct 30 22:53:44.666158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.666108    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-tmp-dir\") pod \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") "
Oct 30 22:53:44.666388 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.666252    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-home" pod "12e366ed-b8f7-4d11-aae0-9f3e4015c8cc" (UID: "12e366ed-b8f7-4d11-aae0-9f3e4015c8cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:44.666388 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.666266    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-tls-certs\") pod \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\" (UID: \"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc\") "
Oct 30 22:53:44.666388 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.666374    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-model-cache" pod "12e366ed-b8f7-4d11-aae0-9f3e4015c8cc" (UID: "12e366ed-b8f7-4d11-aae0-9f3e4015c8cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:44.666535 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.666390    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-tmp-dir" pod "12e366ed-b8f7-4d11-aae0-9f3e4015c8cc" (UID: "12e366ed-b8f7-4d11-aae0-9f3e4015c8cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:44.666676 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.666655    4553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-home\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.666676 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.666676    4553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-model-cache\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.667035 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.666685    4553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-tmp-dir\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.668485 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.668458    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-dshm" pod "12e366ed-b8f7-4d11-aae0-9f3e4015c8cc" (UID: "12e366ed-b8f7-4d11-aae0-9f3e4015c8cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:53:44.668563 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.668549    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-kube-api-access-k8rt8" pod "12e366ed-b8f7-4d11-aae0-9f3e4015c8cc" (UID: "12e366ed-b8f7-4d11-aae0-9f3e4015c8cc"). InnerVolumeSpecName "kube-api-access-k8rt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:44.668610 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.668566    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-tls-certs" pod "12e366ed-b8f7-4d11-aae0-9f3e4015c8cc" (UID: "12e366ed-b8f7-4d11-aae0-9f3e4015c8cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:44.686000 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.685975    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:53:44.686351 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.686025    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh" podUID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 22:53:44.765397 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.765354    4553 generic.go:356] "Generic (PLEG): container finished" podID="12e366ed-b8f7-4d11-aae0-9f3e4015c8cc" containerID="f5db27f72443ee9a4ece8a6991e991d858f5cafae9dfb0234c521875e399cafc" exitCode=0
Oct 30 22:53:44.765607 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.765484    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"
Oct 30 22:53:44.765681 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.765660    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v" event={"ID":"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc","Type":"ContainerDied","Data":"f5db27f72443ee9a4ece8a6991e991d858f5cafae9dfb0234c521875e399cafc"}
Oct 30 22:53:44.765741 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.765706    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v" event={"ID":"12e366ed-b8f7-4d11-aae0-9f3e4015c8cc","Type":"ContainerDied","Data":"1bf43e3bdb3e0d75562aa8f55bc2a7d367dd94695b2595c3d02539d870a3ece3"}
Oct 30 22:53:44.765795 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.765747    4553 scope.go:122] "RemoveContainer" containerID="f5db27f72443ee9a4ece8a6991e991d858f5cafae9dfb0234c521875e399cafc"
Oct 30 22:53:44.769094 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.769063    4553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-dshm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.769360 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.769101    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8rt8\" (UniqueName: \"kubernetes.io/projected/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-kube-api-access-k8rt8\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.769360 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.769117    4553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc-tls-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:44.783196 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.783170    4553 scope.go:122] "RemoveContainer" containerID="f5db27f72443ee9a4ece8a6991e991d858f5cafae9dfb0234c521875e399cafc"
Oct 30 22:53:44.783627 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:53:44.783597    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5db27f72443ee9a4ece8a6991e991d858f5cafae9dfb0234c521875e399cafc\": container with ID starting with f5db27f72443ee9a4ece8a6991e991d858f5cafae9dfb0234c521875e399cafc not found: ID does not exist" containerID="f5db27f72443ee9a4ece8a6991e991d858f5cafae9dfb0234c521875e399cafc"
Oct 30 22:53:44.783745 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.783638    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5db27f72443ee9a4ece8a6991e991d858f5cafae9dfb0234c521875e399cafc"} err="failed to get container status \"f5db27f72443ee9a4ece8a6991e991d858f5cafae9dfb0234c521875e399cafc\": rpc error: code = NotFound desc = could not find container \"f5db27f72443ee9a4ece8a6991e991d858f5cafae9dfb0234c521875e399cafc\": container with ID starting with f5db27f72443ee9a4ece8a6991e991d858f5cafae9dfb0234c521875e399cafc not found: ID does not exist"
Oct 30 22:53:44.945744 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.945646    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"]
Oct 30 22:53:44.951359 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:44.951326    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-90e0d5/distinct-2-w2-90e0d5-kserve-755c6f59bc-6x77v"]
Oct 30 22:53:45.393103 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:45.393070    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e366ed-b8f7-4d11-aae0-9f3e4015c8cc" path="/var/lib/kubelet/pods/12e366ed-b8f7-4d11-aae0-9f3e4015c8cc/volumes"
Oct 30 22:53:45.685213 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:45.685108    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:53:45.685414 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:45.685208    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh" podUID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 22:53:46.685351 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:46.685314    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body=
Oct 30 22:53:46.685822 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:46.685372    4553 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh" podUID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused"
Oct 30 22:53:47.689358 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:47.689321    4553 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:53:47.689844 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:47.689724    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:53:47.690484 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:47.690464    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:53:47.866572 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:47.866537    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"]
Oct 30 22:53:47.866799 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:47.866773    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc" podUID="d71f867b-5b77-4792-b2d9-71d325c58450" containerName="maas-api" containerID="cri-o://53e8fb18d7796d4e45dbc651b4956dc0d5ec173c74e52b8442dd300d2b7339ba" gracePeriod=30
Oct 30 22:53:48.122562 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.122533    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 22:53:48.207682 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.207648    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71f867b-5b77-4792-b2d9-71d325c58450-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71f867b-5b77-4792-b2d9-71d325c58450-openshift-service-ca\") pod \"d71f867b-5b77-4792-b2d9-71d325c58450\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") "
Oct 30 22:53:48.207870 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.207788    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71f867b-5b77-4792-b2d9-71d325c58450-kube-api-access-7sjc7\" (UniqueName: \"kubernetes.io/projected/d71f867b-5b77-4792-b2d9-71d325c58450-kube-api-access-7sjc7\") pod \"d71f867b-5b77-4792-b2d9-71d325c58450\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") "
Oct 30 22:53:48.207914 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.207883    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-metrics-certs\") pod \"d71f867b-5b77-4792-b2d9-71d325c58450\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") "
Oct 30 22:53:48.207958 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.207913    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-maas-api-tls\") pod \"d71f867b-5b77-4792-b2d9-71d325c58450\" (UID: \"d71f867b-5b77-4792-b2d9-71d325c58450\") "
Oct 30 22:53:48.208119 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.208082    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71f867b-5b77-4792-b2d9-71d325c58450-openshift-service-ca" pod "d71f867b-5b77-4792-b2d9-71d325c58450" (UID: "d71f867b-5b77-4792-b2d9-71d325c58450"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:48.208379 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.208360    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d71f867b-5b77-4792-b2d9-71d325c58450-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:48.210295 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.210271    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-maas-api-tls" pod "d71f867b-5b77-4792-b2d9-71d325c58450" (UID: "d71f867b-5b77-4792-b2d9-71d325c58450"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:48.210383 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.210289    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-metrics-certs" pod "d71f867b-5b77-4792-b2d9-71d325c58450" (UID: "d71f867b-5b77-4792-b2d9-71d325c58450"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:48.211241 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.211224    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71f867b-5b77-4792-b2d9-71d325c58450-kube-api-access-7sjc7" pod "d71f867b-5b77-4792-b2d9-71d325c58450" (UID: "d71f867b-5b77-4792-b2d9-71d325c58450"). InnerVolumeSpecName "kube-api-access-7sjc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:48.309525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.309445    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sjc7\" (UniqueName: \"kubernetes.io/projected/d71f867b-5b77-4792-b2d9-71d325c58450-kube-api-access-7sjc7\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:48.309525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.309477    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:48.309525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.309488    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d71f867b-5b77-4792-b2d9-71d325c58450-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:53:48.791200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.791163    4553 generic.go:356] "Generic (PLEG): container finished" podID="d71f867b-5b77-4792-b2d9-71d325c58450" containerID="53e8fb18d7796d4e45dbc651b4956dc0d5ec173c74e52b8442dd300d2b7339ba" exitCode=0
Oct 30 22:53:48.791200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.791200    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"
Oct 30 22:53:48.791811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.791232    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc" event={"ID":"d71f867b-5b77-4792-b2d9-71d325c58450","Type":"ContainerDied","Data":"53e8fb18d7796d4e45dbc651b4956dc0d5ec173c74e52b8442dd300d2b7339ba"}
Oct 30 22:53:48.791811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.791289    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc" event={"ID":"d71f867b-5b77-4792-b2d9-71d325c58450","Type":"ContainerDied","Data":"90739373c75d9476fd094bd4aad3ed3982f70e6c02c1821f8be77d2220810b7a"}
Oct 30 22:53:48.791811 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.791318    4553 scope.go:122] "RemoveContainer" containerID="53e8fb18d7796d4e45dbc651b4956dc0d5ec173c74e52b8442dd300d2b7339ba"
Oct 30 22:53:48.801671 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.801647    4553 scope.go:122] "RemoveContainer" containerID="53e8fb18d7796d4e45dbc651b4956dc0d5ec173c74e52b8442dd300d2b7339ba"
Oct 30 22:53:48.802026 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:53:48.802003    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e8fb18d7796d4e45dbc651b4956dc0d5ec173c74e52b8442dd300d2b7339ba\": container with ID starting with 53e8fb18d7796d4e45dbc651b4956dc0d5ec173c74e52b8442dd300d2b7339ba not found: ID does not exist" containerID="53e8fb18d7796d4e45dbc651b4956dc0d5ec173c74e52b8442dd300d2b7339ba"
Oct 30 22:53:48.802157 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.802033    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e8fb18d7796d4e45dbc651b4956dc0d5ec173c74e52b8442dd300d2b7339ba"} err="failed to get container status \"53e8fb18d7796d4e45dbc651b4956dc0d5ec173c74e52b8442dd300d2b7339ba\": rpc error: code = NotFound desc = could not find container \"53e8fb18d7796d4e45dbc651b4956dc0d5ec173c74e52b8442dd300d2b7339ba\": container with ID starting with 53e8fb18d7796d4e45dbc651b4956dc0d5ec173c74e52b8442dd300d2b7339ba not found: ID does not exist"
Oct 30 22:53:48.957639 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.957600    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"]
Oct 30 22:53:48.961864 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:48.961826    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b8717e-545c8df8fd-7ggxc"]
Oct 30 22:53:49.392965 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:53:49.392934    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71f867b-5b77-4792-b2d9-71d325c58450" path="/var/lib/kubelet/pods/d71f867b-5b77-4792-b2d9-71d325c58450/volumes"
Oct 30 22:53:58.202516 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:53:58.202475    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:54:28.202359 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:54:28.202320    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:54:44.003560 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:44.003522    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"]
Oct 30 22:54:44.111511 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:44.111477    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"]
Oct 30 22:54:49.004440 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.004368    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s" podUID="7103581d-3919-4cb9-bae3-9ade62f0ed1e" containerName="payload-processing" containerID="cri-o://be0bde7b47f4949ec4b1d1021cc0e051e5cf696774b47f1390e6a3d2a5034840" gracePeriod=25
Oct 30 22:54:49.112086 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.112040    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr" podUID="e2b08777-7389-4a3c-acc8-95e0c5a274bd" containerName="payload-pre-processing" containerID="cri-o://2eda2b2c16c6e2ab95ee60e3ce9b3f1353314cf5fde035969891790a26845f08" gracePeriod=25
Oct 30 22:54:49.157484 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.157452    4553 generic.go:356] "Generic (PLEG): container finished" podID="7103581d-3919-4cb9-bae3-9ade62f0ed1e" containerID="be0bde7b47f4949ec4b1d1021cc0e051e5cf696774b47f1390e6a3d2a5034840" exitCode=0
Oct 30 22:54:49.157693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.157505    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s" event={"ID":"7103581d-3919-4cb9-bae3-9ade62f0ed1e","Type":"ContainerDied","Data":"be0bde7b47f4949ec4b1d1021cc0e051e5cf696774b47f1390e6a3d2a5034840"}
Oct 30 22:54:49.259203 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.259091    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"
Oct 30 22:54:49.361526 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.361493    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"
Oct 30 22:54:49.362468 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.362443    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7103581d-3919-4cb9-bae3-9ade62f0ed1e-kube-api-access-lsmzk\" (UniqueName: \"kubernetes.io/projected/7103581d-3919-4cb9-bae3-9ade62f0ed1e-kube-api-access-lsmzk\") pod \"7103581d-3919-4cb9-bae3-9ade62f0ed1e\" (UID: \"7103581d-3919-4cb9-bae3-9ade62f0ed1e\") "
Oct 30 22:54:49.362649 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.362628    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7103581d-3919-4cb9-bae3-9ade62f0ed1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7103581d-3919-4cb9-bae3-9ade62f0ed1e-plugins-config-volume\") pod \"7103581d-3919-4cb9-bae3-9ade62f0ed1e\" (UID: \"7103581d-3919-4cb9-bae3-9ade62f0ed1e\") "
Oct 30 22:54:49.363041 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.363020    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7103581d-3919-4cb9-bae3-9ade62f0ed1e-plugins-config-volume" pod "7103581d-3919-4cb9-bae3-9ade62f0ed1e" (UID: "7103581d-3919-4cb9-bae3-9ade62f0ed1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:49.363108 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.363092    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7103581d-3919-4cb9-bae3-9ade62f0ed1e-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:49.364784 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.364764    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7103581d-3919-4cb9-bae3-9ade62f0ed1e-kube-api-access-lsmzk" pod "7103581d-3919-4cb9-bae3-9ade62f0ed1e" (UID: "7103581d-3919-4cb9-bae3-9ade62f0ed1e"). InnerVolumeSpecName "kube-api-access-lsmzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:49.463611 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.463580    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b08777-7389-4a3c-acc8-95e0c5a274bd-kube-api-access-mrf2w\" (UniqueName: \"kubernetes.io/projected/e2b08777-7389-4a3c-acc8-95e0c5a274bd-kube-api-access-mrf2w\") pod \"e2b08777-7389-4a3c-acc8-95e0c5a274bd\" (UID: \"e2b08777-7389-4a3c-acc8-95e0c5a274bd\") "
Oct 30 22:54:49.463808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.463684    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2b08777-7389-4a3c-acc8-95e0c5a274bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2b08777-7389-4a3c-acc8-95e0c5a274bd-plugins-config-volume\") pod \"e2b08777-7389-4a3c-acc8-95e0c5a274bd\" (UID: \"e2b08777-7389-4a3c-acc8-95e0c5a274bd\") "
Oct 30 22:54:49.464079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.464057    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsmzk\" (UniqueName: \"kubernetes.io/projected/7103581d-3919-4cb9-bae3-9ade62f0ed1e-kube-api-access-lsmzk\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:49.464169 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.464096    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b08777-7389-4a3c-acc8-95e0c5a274bd-plugins-config-volume" pod "e2b08777-7389-4a3c-acc8-95e0c5a274bd" (UID: "e2b08777-7389-4a3c-acc8-95e0c5a274bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:49.465887 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.465860    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b08777-7389-4a3c-acc8-95e0c5a274bd-kube-api-access-mrf2w" pod "e2b08777-7389-4a3c-acc8-95e0c5a274bd" (UID: "e2b08777-7389-4a3c-acc8-95e0c5a274bd"). InnerVolumeSpecName "kube-api-access-mrf2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:49.564773 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.564682    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrf2w\" (UniqueName: \"kubernetes.io/projected/e2b08777-7389-4a3c-acc8-95e0c5a274bd-kube-api-access-mrf2w\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:49.564773 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:49.564715    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2b08777-7389-4a3c-acc8-95e0c5a274bd-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:54:50.165313 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:50.165277    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"
Oct 30 22:54:50.165859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:50.165323    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s" event={"ID":"7103581d-3919-4cb9-bae3-9ade62f0ed1e","Type":"ContainerDied","Data":"e60de3cd334086a60b5672e24fce2bdaee7fe388b836f2060be4bb6fef86765c"}
Oct 30 22:54:50.165859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:50.165407    4553 scope.go:122] "RemoveContainer" containerID="be0bde7b47f4949ec4b1d1021cc0e051e5cf696774b47f1390e6a3d2a5034840"
Oct 30 22:54:50.166890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:50.166869    4553 generic.go:356] "Generic (PLEG): container finished" podID="e2b08777-7389-4a3c-acc8-95e0c5a274bd" containerID="2eda2b2c16c6e2ab95ee60e3ce9b3f1353314cf5fde035969891790a26845f08" exitCode=0
Oct 30 22:54:50.166965 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:50.166915    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"
Oct 30 22:54:50.166965 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:50.166952    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr" event={"ID":"e2b08777-7389-4a3c-acc8-95e0c5a274bd","Type":"ContainerDied","Data":"2eda2b2c16c6e2ab95ee60e3ce9b3f1353314cf5fde035969891790a26845f08"}
Oct 30 22:54:50.167053 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:50.167015    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr" event={"ID":"e2b08777-7389-4a3c-acc8-95e0c5a274bd","Type":"ContainerDied","Data":"b00dc448ca471f3ad8c6f4139834e4cb9a5ee231229bc3a16bc7cf3cd29bb30d"}
Oct 30 22:54:50.175922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:50.175905    4553 scope.go:122] "RemoveContainer" containerID="2eda2b2c16c6e2ab95ee60e3ce9b3f1353314cf5fde035969891790a26845f08"
Oct 30 22:54:50.184734 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:50.184708    4553 scope.go:122] "RemoveContainer" containerID="2eda2b2c16c6e2ab95ee60e3ce9b3f1353314cf5fde035969891790a26845f08"
Oct 30 22:54:50.185016 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:54:50.184995    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eda2b2c16c6e2ab95ee60e3ce9b3f1353314cf5fde035969891790a26845f08\": container with ID starting with 2eda2b2c16c6e2ab95ee60e3ce9b3f1353314cf5fde035969891790a26845f08 not found: ID does not exist" containerID="2eda2b2c16c6e2ab95ee60e3ce9b3f1353314cf5fde035969891790a26845f08"
Oct 30 22:54:50.185087 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:50.185023    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eda2b2c16c6e2ab95ee60e3ce9b3f1353314cf5fde035969891790a26845f08"} err="failed to get container status \"2eda2b2c16c6e2ab95ee60e3ce9b3f1353314cf5fde035969891790a26845f08\": rpc error: code = NotFound desc = could not find container \"2eda2b2c16c6e2ab95ee60e3ce9b3f1353314cf5fde035969891790a26845f08\": container with ID starting with 2eda2b2c16c6e2ab95ee60e3ce9b3f1353314cf5fde035969891790a26845f08 not found: ID does not exist"
Oct 30 22:54:50.336508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:50.336470    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"]
Oct 30 22:54:50.345347 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:50.345318    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6924e5-559df96b8c-mndrr"]
Oct 30 22:54:50.455221 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:50.455189    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"]
Oct 30 22:54:50.459051 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:50.459022    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6924e5-7577ccfff7-qzt9s"]
Oct 30 22:54:51.392370 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:51.392338    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7103581d-3919-4cb9-bae3-9ade62f0ed1e" path="/var/lib/kubelet/pods/7103581d-3919-4cb9-bae3-9ade62f0ed1e/volumes"
Oct 30 22:54:51.392764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:54:51.392718    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2b08777-7389-4a3c-acc8-95e0c5a274bd" path="/var/lib/kubelet/pods/e2b08777-7389-4a3c-acc8-95e0c5a274bd/volumes"
Oct 30 22:54:58.200782 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:54:58.200746    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:55:05.177908 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:05.177870    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"]
Oct 30 22:55:05.178466 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:05.178088    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp" podUID="c4d6c182-7bf5-4a2c-95d5-174b4f34d298" containerName="maas-api" containerID="cri-o://c7458f2586909bbced7745efe43f8219abe4739f3056f5d330faa1fa5c0eafac" gracePeriod=30
Oct 30 22:55:05.455453 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:05.455423    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:55:05.543581 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:05.543548    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-openshift-service-ca\") pod \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") "
Oct 30 22:55:05.543802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:05.543617    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-kube-api-access-cf255\" (UniqueName: \"kubernetes.io/projected/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-kube-api-access-cf255\") pod \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") "
Oct 30 22:55:05.543802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:05.543628    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-openshift-service-ca" pod "c4d6c182-7bf5-4a2c-95d5-174b4f34d298" (UID: "c4d6c182-7bf5-4a2c-95d5-174b4f34d298"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:05.543802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:05.543649    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-metrics-certs\") pod \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") "
Oct 30 22:55:05.543802 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:05.543769    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-maas-api-tls\") pod \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\" (UID: \"c4d6c182-7bf5-4a2c-95d5-174b4f34d298\") "
Oct 30 22:55:05.544125 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:05.544106    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:05.553915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:05.546620    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-kube-api-access-cf255" pod "c4d6c182-7bf5-4a2c-95d5-174b4f34d298" (UID: "c4d6c182-7bf5-4a2c-95d5-174b4f34d298"). InnerVolumeSpecName "kube-api-access-cf255". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:05.553915 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:05.546763    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-metrics-certs" pod "c4d6c182-7bf5-4a2c-95d5-174b4f34d298" (UID: "c4d6c182-7bf5-4a2c-95d5-174b4f34d298"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:05.554772 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:05.554733    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-maas-api-tls" pod "c4d6c182-7bf5-4a2c-95d5-174b4f34d298" (UID: "c4d6c182-7bf5-4a2c-95d5-174b4f34d298"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:05.644859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:05.644820    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf255\" (UniqueName: \"kubernetes.io/projected/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-kube-api-access-cf255\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:05.644859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:05.644852    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:05.644859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:05.644861    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4d6c182-7bf5-4a2c-95d5-174b4f34d298-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:06.255019 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:06.254984    4553 generic.go:356] "Generic (PLEG): container finished" podID="c4d6c182-7bf5-4a2c-95d5-174b4f34d298" containerID="c7458f2586909bbced7745efe43f8219abe4739f3056f5d330faa1fa5c0eafac" exitCode=0
Oct 30 22:55:06.255478 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:06.255045    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"
Oct 30 22:55:06.255478 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:06.255064    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp" event={"ID":"c4d6c182-7bf5-4a2c-95d5-174b4f34d298","Type":"ContainerDied","Data":"c7458f2586909bbced7745efe43f8219abe4739f3056f5d330faa1fa5c0eafac"}
Oct 30 22:55:06.255478 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:06.255158    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp" event={"ID":"c4d6c182-7bf5-4a2c-95d5-174b4f34d298","Type":"ContainerDied","Data":"51d2a9e93841d4bd7748b8a79f7e0384e5d6c0807662f7b0c5c80d5efa8dafb6"}
Oct 30 22:55:06.255478 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:06.255184    4553 scope.go:122] "RemoveContainer" containerID="c7458f2586909bbced7745efe43f8219abe4739f3056f5d330faa1fa5c0eafac"
Oct 30 22:55:06.265285 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:06.265264    4553 scope.go:122] "RemoveContainer" containerID="c7458f2586909bbced7745efe43f8219abe4739f3056f5d330faa1fa5c0eafac"
Oct 30 22:55:06.265540 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:55:06.265520    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7458f2586909bbced7745efe43f8219abe4739f3056f5d330faa1fa5c0eafac\": container with ID starting with c7458f2586909bbced7745efe43f8219abe4739f3056f5d330faa1fa5c0eafac not found: ID does not exist" containerID="c7458f2586909bbced7745efe43f8219abe4739f3056f5d330faa1fa5c0eafac"
Oct 30 22:55:06.265593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:06.265551    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7458f2586909bbced7745efe43f8219abe4739f3056f5d330faa1fa5c0eafac"} err="failed to get container status \"c7458f2586909bbced7745efe43f8219abe4739f3056f5d330faa1fa5c0eafac\": rpc error: code = NotFound desc = could not find container \"c7458f2586909bbced7745efe43f8219abe4739f3056f5d330faa1fa5c0eafac\": container with ID starting with c7458f2586909bbced7745efe43f8219abe4739f3056f5d330faa1fa5c0eafac not found: ID does not exist"
Oct 30 22:55:06.424351 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:06.424313    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"]
Oct 30 22:55:06.427352 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:06.427326    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6924e5-6858df65f8-4sjjp"]
Oct 30 22:55:07.392338 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:07.392302    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4d6c182-7bf5-4a2c-95d5-174b4f34d298" path="/var/lib/kubelet/pods/c4d6c182-7bf5-4a2c-95d5-174b4f34d298/volumes"
Oct 30 22:55:28.201153 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:55:28.201095    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:55:46.180466 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.180428    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"]
Oct 30 22:55:46.180957 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.180660    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4" podUID="62d53659-7ffd-4b08-ae37-8e705017b7d9" containerName="maas-api" containerID="cri-o://ba1f5d5dc8ebabd4a47453508d6b5214e3d8a9c6d4b17647e460e9d058352bb1" gracePeriod=30
Oct 30 22:55:46.430748 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.430679    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:55:46.464696 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.464667    4553 generic.go:356] "Generic (PLEG): container finished" podID="62d53659-7ffd-4b08-ae37-8e705017b7d9" containerID="ba1f5d5dc8ebabd4a47453508d6b5214e3d8a9c6d4b17647e460e9d058352bb1" exitCode=0
Oct 30 22:55:46.464860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.464715    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"
Oct 30 22:55:46.464860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.464755    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4" event={"ID":"62d53659-7ffd-4b08-ae37-8e705017b7d9","Type":"ContainerDied","Data":"ba1f5d5dc8ebabd4a47453508d6b5214e3d8a9c6d4b17647e460e9d058352bb1"}
Oct 30 22:55:46.464860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.464814    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4" event={"ID":"62d53659-7ffd-4b08-ae37-8e705017b7d9","Type":"ContainerDied","Data":"74b718ddab37b34c9a1892e7853209b20c3f76162692819af9af6b60479ed7e2"}
Oct 30 22:55:46.464860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.464844    4553 scope.go:122] "RemoveContainer" containerID="ba1f5d5dc8ebabd4a47453508d6b5214e3d8a9c6d4b17647e460e9d058352bb1"
Oct 30 22:55:46.474001 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.473976    4553 scope.go:122] "RemoveContainer" containerID="ba1f5d5dc8ebabd4a47453508d6b5214e3d8a9c6d4b17647e460e9d058352bb1"
Oct 30 22:55:46.474311 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:55:46.474286    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1f5d5dc8ebabd4a47453508d6b5214e3d8a9c6d4b17647e460e9d058352bb1\": container with ID starting with ba1f5d5dc8ebabd4a47453508d6b5214e3d8a9c6d4b17647e460e9d058352bb1 not found: ID does not exist" containerID="ba1f5d5dc8ebabd4a47453508d6b5214e3d8a9c6d4b17647e460e9d058352bb1"
Oct 30 22:55:46.474397 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.474323    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1f5d5dc8ebabd4a47453508d6b5214e3d8a9c6d4b17647e460e9d058352bb1"} err="failed to get container status \"ba1f5d5dc8ebabd4a47453508d6b5214e3d8a9c6d4b17647e460e9d058352bb1\": rpc error: code = NotFound desc = could not find container \"ba1f5d5dc8ebabd4a47453508d6b5214e3d8a9c6d4b17647e460e9d058352bb1\": container with ID starting with ba1f5d5dc8ebabd4a47453508d6b5214e3d8a9c6d4b17647e460e9d058352bb1 not found: ID does not exist"
Oct 30 22:55:46.506031 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.505995    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"]
Oct 30 22:55:46.590035 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.589994    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-metrics-certs\") pod \"62d53659-7ffd-4b08-ae37-8e705017b7d9\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") "
Oct 30 22:55:46.590250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.590061    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d53659-7ffd-4b08-ae37-8e705017b7d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d53659-7ffd-4b08-ae37-8e705017b7d9-openshift-service-ca\") pod \"62d53659-7ffd-4b08-ae37-8e705017b7d9\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") "
Oct 30 22:55:46.590250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.590117    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d53659-7ffd-4b08-ae37-8e705017b7d9-kube-api-access-pbtw5\" (UniqueName: \"kubernetes.io/projected/62d53659-7ffd-4b08-ae37-8e705017b7d9-kube-api-access-pbtw5\") pod \"62d53659-7ffd-4b08-ae37-8e705017b7d9\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") "
Oct 30 22:55:46.590250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.590165    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-maas-api-tls\") pod \"62d53659-7ffd-4b08-ae37-8e705017b7d9\" (UID: \"62d53659-7ffd-4b08-ae37-8e705017b7d9\") "
Oct 30 22:55:46.590537 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.590505    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d53659-7ffd-4b08-ae37-8e705017b7d9-openshift-service-ca" pod "62d53659-7ffd-4b08-ae37-8e705017b7d9" (UID: "62d53659-7ffd-4b08-ae37-8e705017b7d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:46.592329 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.592300    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-metrics-certs" pod "62d53659-7ffd-4b08-ae37-8e705017b7d9" (UID: "62d53659-7ffd-4b08-ae37-8e705017b7d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:46.592429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.592333    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d53659-7ffd-4b08-ae37-8e705017b7d9-kube-api-access-pbtw5" pod "62d53659-7ffd-4b08-ae37-8e705017b7d9" (UID: "62d53659-7ffd-4b08-ae37-8e705017b7d9"). InnerVolumeSpecName "kube-api-access-pbtw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:46.592429 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.592412    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-maas-api-tls" pod "62d53659-7ffd-4b08-ae37-8e705017b7d9" (UID: "62d53659-7ffd-4b08-ae37-8e705017b7d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:55:46.640403 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.640365    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"]
Oct 30 22:55:46.691115 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.691038    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:46.691115 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.691063    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d53659-7ffd-4b08-ae37-8e705017b7d9-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:46.691115 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.691074    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbtw5\" (UniqueName: \"kubernetes.io/projected/62d53659-7ffd-4b08-ae37-8e705017b7d9-kube-api-access-pbtw5\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:46.691115 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.691083    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62d53659-7ffd-4b08-ae37-8e705017b7d9-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:46.953625 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.953533    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"]
Oct 30 22:55:46.957446 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:46.957415    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3faf84-7b9b6d89c4-qdgc4"]
Oct 30 22:55:47.392500 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:47.392469    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62d53659-7ffd-4b08-ae37-8e705017b7d9" path="/var/lib/kubelet/pods/62d53659-7ffd-4b08-ae37-8e705017b7d9/volumes"
Oct 30 22:55:51.507015 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:51.506923    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r" podUID="0eaa3e7a-052e-4f6a-9cd3-d3a648528160" containerName="payload-processing" containerID="cri-o://b47e3ac64d60033222669276d6870928f16f14c241c8b8772bcba1b5d16c1add" gracePeriod=25
Oct 30 22:55:51.640806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:51.640759    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n" podUID="c065085c-3963-45a0-b4bc-09ad435e27b7" containerName="payload-pre-processing" containerID="cri-o://dda1ba985345bc4262a1cfe74f53e08c79e6b79960a591014203e9c6e3f50cc5" gracePeriod=25
Oct 30 22:55:51.779727 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:51.772459    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"
Oct 30 22:55:51.840421 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:51.840387    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eaa3e7a-052e-4f6a-9cd3-d3a648528160-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eaa3e7a-052e-4f6a-9cd3-d3a648528160-plugins-config-volume\") pod \"0eaa3e7a-052e-4f6a-9cd3-d3a648528160\" (UID: \"0eaa3e7a-052e-4f6a-9cd3-d3a648528160\") "
Oct 30 22:55:51.840631 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:51.840517    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eaa3e7a-052e-4f6a-9cd3-d3a648528160-kube-api-access-nf5sq\" (UniqueName: \"kubernetes.io/projected/0eaa3e7a-052e-4f6a-9cd3-d3a648528160-kube-api-access-nf5sq\") pod \"0eaa3e7a-052e-4f6a-9cd3-d3a648528160\" (UID: \"0eaa3e7a-052e-4f6a-9cd3-d3a648528160\") "
Oct 30 22:55:51.841498 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:51.841450    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eaa3e7a-052e-4f6a-9cd3-d3a648528160-plugins-config-volume" pod "0eaa3e7a-052e-4f6a-9cd3-d3a648528160" (UID: "0eaa3e7a-052e-4f6a-9cd3-d3a648528160"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:51.850408 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:51.843682    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eaa3e7a-052e-4f6a-9cd3-d3a648528160-kube-api-access-nf5sq" pod "0eaa3e7a-052e-4f6a-9cd3-d3a648528160" (UID: "0eaa3e7a-052e-4f6a-9cd3-d3a648528160"). InnerVolumeSpecName "kube-api-access-nf5sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:51.889633 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:51.889604    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"
Oct 30 22:55:51.941684 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:51.941652    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c065085c-3963-45a0-b4bc-09ad435e27b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c065085c-3963-45a0-b4bc-09ad435e27b7-plugins-config-volume\") pod \"c065085c-3963-45a0-b4bc-09ad435e27b7\" (UID: \"c065085c-3963-45a0-b4bc-09ad435e27b7\") "
Oct 30 22:55:51.941859 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:51.941760    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c065085c-3963-45a0-b4bc-09ad435e27b7-kube-api-access-dxxlm\" (UniqueName: \"kubernetes.io/projected/c065085c-3963-45a0-b4bc-09ad435e27b7-kube-api-access-dxxlm\") pod \"c065085c-3963-45a0-b4bc-09ad435e27b7\" (UID: \"c065085c-3963-45a0-b4bc-09ad435e27b7\") "
Oct 30 22:55:51.942079 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:51.942046    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c065085c-3963-45a0-b4bc-09ad435e27b7-plugins-config-volume" pod "c065085c-3963-45a0-b4bc-09ad435e27b7" (UID: "c065085c-3963-45a0-b4bc-09ad435e27b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:51.942250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:51.942200    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf5sq\" (UniqueName: \"kubernetes.io/projected/0eaa3e7a-052e-4f6a-9cd3-d3a648528160-kube-api-access-nf5sq\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:51.942250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:51.942216    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c065085c-3963-45a0-b4bc-09ad435e27b7-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:51.942250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:51.942229    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eaa3e7a-052e-4f6a-9cd3-d3a648528160-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:51.944372 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:51.944348    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c065085c-3963-45a0-b4bc-09ad435e27b7-kube-api-access-dxxlm" pod "c065085c-3963-45a0-b4bc-09ad435e27b7" (UID: "c065085c-3963-45a0-b4bc-09ad435e27b7"). InnerVolumeSpecName "kube-api-access-dxxlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:52.043495 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.043412    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxxlm\" (UniqueName: \"kubernetes.io/projected/c065085c-3963-45a0-b4bc-09ad435e27b7-kube-api-access-dxxlm\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:55:52.499442 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.499403    4553 generic.go:356] "Generic (PLEG): container finished" podID="0eaa3e7a-052e-4f6a-9cd3-d3a648528160" containerID="b47e3ac64d60033222669276d6870928f16f14c241c8b8772bcba1b5d16c1add" exitCode=0
Oct 30 22:55:52.499669 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.499458    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"
Oct 30 22:55:52.499669 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.499486    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r" event={"ID":"0eaa3e7a-052e-4f6a-9cd3-d3a648528160","Type":"ContainerDied","Data":"b47e3ac64d60033222669276d6870928f16f14c241c8b8772bcba1b5d16c1add"}
Oct 30 22:55:52.499669 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.499540    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r" event={"ID":"0eaa3e7a-052e-4f6a-9cd3-d3a648528160","Type":"ContainerDied","Data":"abf095c4a6012755cc1a0817bb1d46c150d407653f16c70dd43f7da9d1113511"}
Oct 30 22:55:52.499669 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.499563    4553 scope.go:122] "RemoveContainer" containerID="b47e3ac64d60033222669276d6870928f16f14c241c8b8772bcba1b5d16c1add"
Oct 30 22:55:52.501074 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.501046    4553 generic.go:356] "Generic (PLEG): container finished" podID="c065085c-3963-45a0-b4bc-09ad435e27b7" containerID="dda1ba985345bc4262a1cfe74f53e08c79e6b79960a591014203e9c6e3f50cc5" exitCode=0
Oct 30 22:55:52.501225 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.501095    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n" event={"ID":"c065085c-3963-45a0-b4bc-09ad435e27b7","Type":"ContainerDied","Data":"dda1ba985345bc4262a1cfe74f53e08c79e6b79960a591014203e9c6e3f50cc5"}
Oct 30 22:55:52.501225 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.501104    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"
Oct 30 22:55:52.501225 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.501189    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n" event={"ID":"c065085c-3963-45a0-b4bc-09ad435e27b7","Type":"ContainerDied","Data":"cf650300e6dcc0a5ec750cfcc6505bc922bf486e44ee7adeea7cc726f7059b83"}
Oct 30 22:55:52.512499 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.512403    4553 scope.go:122] "RemoveContainer" containerID="b47e3ac64d60033222669276d6870928f16f14c241c8b8772bcba1b5d16c1add"
Oct 30 22:55:52.512777 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:55:52.512693    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47e3ac64d60033222669276d6870928f16f14c241c8b8772bcba1b5d16c1add\": container with ID starting with b47e3ac64d60033222669276d6870928f16f14c241c8b8772bcba1b5d16c1add not found: ID does not exist" containerID="b47e3ac64d60033222669276d6870928f16f14c241c8b8772bcba1b5d16c1add"
Oct 30 22:55:52.512777 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.512718    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47e3ac64d60033222669276d6870928f16f14c241c8b8772bcba1b5d16c1add"} err="failed to get container status \"b47e3ac64d60033222669276d6870928f16f14c241c8b8772bcba1b5d16c1add\": rpc error: code = NotFound desc = could not find container \"b47e3ac64d60033222669276d6870928f16f14c241c8b8772bcba1b5d16c1add\": container with ID starting with b47e3ac64d60033222669276d6870928f16f14c241c8b8772bcba1b5d16c1add not found: ID does not exist"
Oct 30 22:55:52.512777 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.512735    4553 scope.go:122] "RemoveContainer" containerID="dda1ba985345bc4262a1cfe74f53e08c79e6b79960a591014203e9c6e3f50cc5"
Oct 30 22:55:52.531808 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.531788    4553 scope.go:122] "RemoveContainer" containerID="dda1ba985345bc4262a1cfe74f53e08c79e6b79960a591014203e9c6e3f50cc5"
Oct 30 22:55:52.532115 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:55:52.532091    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda1ba985345bc4262a1cfe74f53e08c79e6b79960a591014203e9c6e3f50cc5\": container with ID starting with dda1ba985345bc4262a1cfe74f53e08c79e6b79960a591014203e9c6e3f50cc5 not found: ID does not exist" containerID="dda1ba985345bc4262a1cfe74f53e08c79e6b79960a591014203e9c6e3f50cc5"
Oct 30 22:55:52.532250 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.532120    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda1ba985345bc4262a1cfe74f53e08c79e6b79960a591014203e9c6e3f50cc5"} err="failed to get container status \"dda1ba985345bc4262a1cfe74f53e08c79e6b79960a591014203e9c6e3f50cc5\": rpc error: code = NotFound desc = could not find container \"dda1ba985345bc4262a1cfe74f53e08c79e6b79960a591014203e9c6e3f50cc5\": container with ID starting with dda1ba985345bc4262a1cfe74f53e08c79e6b79960a591014203e9c6e3f50cc5 not found: ID does not exist"
Oct 30 22:55:52.664735 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.664697    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"]
Oct 30 22:55:52.668275 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.668248    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-3faf84-84b85d9c6c-xrj6n"]
Oct 30 22:55:52.779504 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.779418    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"]
Oct 30 22:55:52.783711 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:52.783681    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-3faf84-7bb789f6fc-bhk6r"]
Oct 30 22:55:53.391783 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:53.391750    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eaa3e7a-052e-4f6a-9cd3-d3a648528160" path="/var/lib/kubelet/pods/0eaa3e7a-052e-4f6a-9cd3-d3a648528160/volumes"
Oct 30 22:55:53.392170 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:55:53.392156    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c065085c-3963-45a0-b4bc-09ad435e27b7" path="/var/lib/kubelet/pods/c065085c-3963-45a0-b4bc-09ad435e27b7/volumes"
Oct 30 22:55:58.199752 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:55:58.199715    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:56:00.435557 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:00.435518    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"]
Oct 30 22:56:00.436011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:00.435756    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh" podUID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerName="istio-proxy" containerID="cri-o://9ddb0867779410ab41c3cec7040f2092748b02f4d3562f712424257bc7374fea" gracePeriod=30
Oct 30 22:56:02.692517 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:02.692476    4553 patch_prober.go:29] interesting pod/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:56:02.693021 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:02.692543    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh" podUID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:56:05.569314 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.569284    4553 generic.go:356] "Generic (PLEG): container finished" podID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerID="9ddb0867779410ab41c3cec7040f2092748b02f4d3562f712424257bc7374fea" exitCode=0
Oct 30 22:56:05.569725 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.569359    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh" event={"ID":"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b","Type":"ContainerDied","Data":"9ddb0867779410ab41c3cec7040f2092748b02f4d3562f712424257bc7374fea"}
Oct 30 22:56:05.569725 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.569439    4553 scope.go:122] "RemoveContainer" containerID="3e70ceac94de7789efc6e17df2f2d46f661a842be52c3316f7017e9485b0e487"
Oct 30 22:56:05.700468 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.700402    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:56:05.787468 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.787434    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-data\") pod \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") "
Oct 30 22:56:05.787667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.787488    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-token\" (UniqueName: \"kubernetes.io/projected/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-token\") pod \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") "
Oct 30 22:56:05.787667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.787543    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istiod-ca-cert\") pod \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") "
Oct 30 22:56:05.787667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.787561    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-workload-socket\") pod \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") "
Oct 30 22:56:05.787667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.787594    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-podinfo\") pod \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") "
Oct 30 22:56:05.787667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.787662    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-credential-socket\") pod \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") "
Oct 30 22:56:05.787922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.787715    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-workload-certs\") pod \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") "
Oct 30 22:56:05.787922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.787742    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-kube-api-access-8nvnh\" (UniqueName: \"kubernetes.io/projected/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-kube-api-access-8nvnh\") pod \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") "
Oct 30 22:56:05.787922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.787738    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-data" pod "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" (UID: "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:05.787922 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.787800    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-envoy\") pod \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\" (UID: \"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b\") "
Oct 30 22:56:05.788198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.787986    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-workload-socket" pod "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" (UID: "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:05.788198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.788016    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-workload-certs" pod "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" (UID: "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:05.788198 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.788063    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istiod-ca-cert" pod "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" (UID: "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:05.788378 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.788261    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:05.788378 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.788259    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-credential-socket" pod "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" (UID: "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:05.788378 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.788275    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:05.788378 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.788285    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:05.788378 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.788294    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:05.789960 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.789935    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-token" pod "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" (UID: "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:05.790245 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.790219    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-envoy" pod "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" (UID: "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:05.790584 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.790565    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-podinfo" pod "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" (UID: "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:56:05.790650 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.790601    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-kube-api-access-8nvnh" pod "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" (UID: "6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b"). InnerVolumeSpecName "kube-api-access-8nvnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:05.889161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.889104    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:05.889161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.889146    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:05.889161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.889158    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nvnh\" (UniqueName: \"kubernetes.io/projected/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-kube-api-access-8nvnh\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:05.889161 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.889167    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:05.889451 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:05.889176    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:06.575544 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:06.575514    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"
Oct 30 22:56:06.576003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:06.575514    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh" event={"ID":"6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b","Type":"ContainerDied","Data":"ef117c040ac0dcd00433a38bcfbb5f6cf0508269385afb531f4588100313e80a"}
Oct 30 22:56:06.576003 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:06.575659    4553 scope.go:122] "RemoveContainer" containerID="9ddb0867779410ab41c3cec7040f2092748b02f4d3562f712424257bc7374fea"
Oct 30 22:56:06.793244 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:06.793202    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"]
Oct 30 22:56:06.798800 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:06.798769    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-3faf84-openshift-default-58b4df664f-d4dbh"]
Oct 30 22:56:07.392475 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:07.392434    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" path="/var/lib/kubelet/pods/6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b/volumes"
Oct 30 22:56:12.706093 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:12.706053    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"]
Oct 30 22:56:17.706985 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:17.706913    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r" podUID="049256e6-63e4-4ae9-9943-5c03d0091451" containerName="payload-pre-processing" containerID="cri-o://c0ab0b1cde00524edfd4f0c2ff7826f7952c46290de88120942597dc0e5fc1aa" gracePeriod=25
Oct 30 22:56:19.095353 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:19.095312    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r" podUID="049256e6-63e4-4ae9-9943-5c03d0091451" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:24.189340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:24.189300    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"]
Oct 30 22:56:24.189879 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:24.189619    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn" podUID="51d3cb4a-13d1-4198-ab0e-0cf97376e53f" containerName="istio-proxy" containerID="cri-o://507c3f28caf6a2ac3426785fdb09c82de19d45ccb36d0753d1d3098b23dbe300" gracePeriod=30
Oct 30 22:56:28.200851 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:56:28.200814    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:56:29.095358 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.095321    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r" podUID="049256e6-63e4-4ae9-9943-5c03d0091451" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:29.459453 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.459425    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:56:29.627750 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.627640    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-credential-socket\") pod \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") "
Oct 30 22:56:29.627928 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.627758    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-workload-certs\") pod \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") "
Oct 30 22:56:29.627928 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.627817    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-kube-api-access-2d99s\" (UniqueName: \"kubernetes.io/projected/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-kube-api-access-2d99s\") pod \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") "
Oct 30 22:56:29.627928 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.627858    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-token\" (UniqueName: \"kubernetes.io/projected/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-token\") pod \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") "
Oct 30 22:56:29.627928 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.627899    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istiod-ca-cert\") pod \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") "
Oct 30 22:56:29.628202 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.627939    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-data\") pod \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") "
Oct 30 22:56:29.628202 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.627977    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-envoy\") pod \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") "
Oct 30 22:56:29.628202 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.628021    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-podinfo\") pod \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") "
Oct 30 22:56:29.628202 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.628046    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-credential-socket" pod "51d3cb4a-13d1-4198-ab0e-0cf97376e53f" (UID: "51d3cb4a-13d1-4198-ab0e-0cf97376e53f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:29.628202 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.628073    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-workload-socket\") pod \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\" (UID: \"51d3cb4a-13d1-4198-ab0e-0cf97376e53f\") "
Oct 30 22:56:29.628550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.628201    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-workload-certs" pod "51d3cb4a-13d1-4198-ab0e-0cf97376e53f" (UID: "51d3cb4a-13d1-4198-ab0e-0cf97376e53f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:29.628550 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.628364    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-data" pod "51d3cb4a-13d1-4198-ab0e-0cf97376e53f" (UID: "51d3cb4a-13d1-4198-ab0e-0cf97376e53f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:29.628649 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.628568    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-workload-socket" pod "51d3cb4a-13d1-4198-ab0e-0cf97376e53f" (UID: "51d3cb4a-13d1-4198-ab0e-0cf97376e53f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:29.628649 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.628579    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:29.628649 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.628597    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:29.628649 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.628609    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:29.628914 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.628873    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istiod-ca-cert" pod "51d3cb4a-13d1-4198-ab0e-0cf97376e53f" (UID: "51d3cb4a-13d1-4198-ab0e-0cf97376e53f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:29.630642 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.630619    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-kube-api-access-2d99s" pod "51d3cb4a-13d1-4198-ab0e-0cf97376e53f" (UID: "51d3cb4a-13d1-4198-ab0e-0cf97376e53f"). InnerVolumeSpecName "kube-api-access-2d99s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:29.630990 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.630963    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-token" pod "51d3cb4a-13d1-4198-ab0e-0cf97376e53f" (UID: "51d3cb4a-13d1-4198-ab0e-0cf97376e53f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:29.631098 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.631034    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-envoy" pod "51d3cb4a-13d1-4198-ab0e-0cf97376e53f" (UID: "51d3cb4a-13d1-4198-ab0e-0cf97376e53f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:29.632525 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.632505    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-podinfo" pod "51d3cb4a-13d1-4198-ab0e-0cf97376e53f" (UID: "51d3cb4a-13d1-4198-ab0e-0cf97376e53f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:56:29.694071 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.694036    4553 generic.go:356] "Generic (PLEG): container finished" podID="51d3cb4a-13d1-4198-ab0e-0cf97376e53f" containerID="507c3f28caf6a2ac3426785fdb09c82de19d45ccb36d0753d1d3098b23dbe300" exitCode=0
Oct 30 22:56:29.694293 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.694103    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn" event={"ID":"51d3cb4a-13d1-4198-ab0e-0cf97376e53f","Type":"ContainerDied","Data":"507c3f28caf6a2ac3426785fdb09c82de19d45ccb36d0753d1d3098b23dbe300"}
Oct 30 22:56:29.694293 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.694107    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"
Oct 30 22:56:29.694293 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.694186    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn" event={"ID":"51d3cb4a-13d1-4198-ab0e-0cf97376e53f","Type":"ContainerDied","Data":"de6baa435c002b3c5e1576c9b011103f77f877eb21348386b7955c02b6428d6f"}
Oct 30 22:56:29.694293 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.694218    4553 scope.go:122] "RemoveContainer" containerID="507c3f28caf6a2ac3426785fdb09c82de19d45ccb36d0753d1d3098b23dbe300"
Oct 30 22:56:29.704358 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.704332    4553 scope.go:122] "RemoveContainer" containerID="8f38fa8af5fcc870c9d03f8fbee64f101bc37b64b3ac9a8cb20a1838332936c5"
Oct 30 22:56:29.713075 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.713053    4553 scope.go:122] "RemoveContainer" containerID="507c3f28caf6a2ac3426785fdb09c82de19d45ccb36d0753d1d3098b23dbe300"
Oct 30 22:56:29.713356 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:56:29.713338    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507c3f28caf6a2ac3426785fdb09c82de19d45ccb36d0753d1d3098b23dbe300\": container with ID starting with 507c3f28caf6a2ac3426785fdb09c82de19d45ccb36d0753d1d3098b23dbe300 not found: ID does not exist" containerID="507c3f28caf6a2ac3426785fdb09c82de19d45ccb36d0753d1d3098b23dbe300"
Oct 30 22:56:29.713423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.713365    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507c3f28caf6a2ac3426785fdb09c82de19d45ccb36d0753d1d3098b23dbe300"} err="failed to get container status \"507c3f28caf6a2ac3426785fdb09c82de19d45ccb36d0753d1d3098b23dbe300\": rpc error: code = NotFound desc = could not find container \"507c3f28caf6a2ac3426785fdb09c82de19d45ccb36d0753d1d3098b23dbe300\": container with ID starting with 507c3f28caf6a2ac3426785fdb09c82de19d45ccb36d0753d1d3098b23dbe300 not found: ID does not exist"
Oct 30 22:56:29.713423 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.713383    4553 scope.go:122] "RemoveContainer" containerID="8f38fa8af5fcc870c9d03f8fbee64f101bc37b64b3ac9a8cb20a1838332936c5"
Oct 30 22:56:29.713647 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:56:29.713627    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f38fa8af5fcc870c9d03f8fbee64f101bc37b64b3ac9a8cb20a1838332936c5\": container with ID starting with 8f38fa8af5fcc870c9d03f8fbee64f101bc37b64b3ac9a8cb20a1838332936c5 not found: ID does not exist" containerID="8f38fa8af5fcc870c9d03f8fbee64f101bc37b64b3ac9a8cb20a1838332936c5"
Oct 30 22:56:29.713706 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.713659    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f38fa8af5fcc870c9d03f8fbee64f101bc37b64b3ac9a8cb20a1838332936c5"} err="failed to get container status \"8f38fa8af5fcc870c9d03f8fbee64f101bc37b64b3ac9a8cb20a1838332936c5\": rpc error: code = NotFound desc = could not find container \"8f38fa8af5fcc870c9d03f8fbee64f101bc37b64b3ac9a8cb20a1838332936c5\": container with ID starting with 8f38fa8af5fcc870c9d03f8fbee64f101bc37b64b3ac9a8cb20a1838332936c5 not found: ID does not exist"
Oct 30 22:56:29.729918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.729890    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d99s\" (UniqueName: \"kubernetes.io/projected/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-kube-api-access-2d99s\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:29.729918 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.729921    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:29.730090 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.729934    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:29.730090 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.729946    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:29.730090 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.729958    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:29.730090 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.729969    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d3cb4a-13d1-4198-ab0e-0cf97376e53f-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:29.865714 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.865680    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"]
Oct 30 22:56:29.869475 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:29.869444    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-89cf28-openshift-default-6f6c4ff5c7-d2lbn"]
Oct 30 22:56:31.392799 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:31.392763    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d3cb4a-13d1-4198-ab0e-0cf97376e53f" path="/var/lib/kubelet/pods/51d3cb4a-13d1-4198-ab0e-0cf97376e53f/volumes"
Oct 30 22:56:36.549566 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:36.549527    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"]
Oct 30 22:56:36.658484 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:36.658452    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"]
Oct 30 22:56:37.966282 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:37.966210    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"
Oct 30 22:56:37.995477 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:37.995441    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049256e6-63e4-4ae9-9943-5c03d0091451-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049256e6-63e4-4ae9-9943-5c03d0091451-plugins-config-volume\") pod \"049256e6-63e4-4ae9-9943-5c03d0091451\" (UID: \"049256e6-63e4-4ae9-9943-5c03d0091451\") "
Oct 30 22:56:37.995667 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:37.995577    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049256e6-63e4-4ae9-9943-5c03d0091451-kube-api-access-mnq9t\" (UniqueName: \"kubernetes.io/projected/049256e6-63e4-4ae9-9943-5c03d0091451-kube-api-access-mnq9t\") pod \"049256e6-63e4-4ae9-9943-5c03d0091451\" (UID: \"049256e6-63e4-4ae9-9943-5c03d0091451\") "
Oct 30 22:56:37.995834 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:37.995809    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049256e6-63e4-4ae9-9943-5c03d0091451-plugins-config-volume" pod "049256e6-63e4-4ae9-9943-5c03d0091451" (UID: "049256e6-63e4-4ae9-9943-5c03d0091451"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:37.997814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:37.997792    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049256e6-63e4-4ae9-9943-5c03d0091451-kube-api-access-mnq9t" pod "049256e6-63e4-4ae9-9943-5c03d0091451" (UID: "049256e6-63e4-4ae9-9943-5c03d0091451"). InnerVolumeSpecName "kube-api-access-mnq9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:38.097063 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:38.097026    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnq9t\" (UniqueName: \"kubernetes.io/projected/049256e6-63e4-4ae9-9943-5c03d0091451-kube-api-access-mnq9t\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:38.097063 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:38.097060    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049256e6-63e4-4ae9-9943-5c03d0091451-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:38.741313 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:38.741281    4553 generic.go:356] "Generic (PLEG): container finished" podID="049256e6-63e4-4ae9-9943-5c03d0091451" containerID="c0ab0b1cde00524edfd4f0c2ff7826f7952c46290de88120942597dc0e5fc1aa" exitCode=0
Oct 30 22:56:38.741534 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:38.741332    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"
Oct 30 22:56:38.741534 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:38.741366    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r" event={"ID":"049256e6-63e4-4ae9-9943-5c03d0091451","Type":"ContainerDied","Data":"c0ab0b1cde00524edfd4f0c2ff7826f7952c46290de88120942597dc0e5fc1aa"}
Oct 30 22:56:38.741534 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:38.741431    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r" event={"ID":"049256e6-63e4-4ae9-9943-5c03d0091451","Type":"ContainerDied","Data":"f12ce661ed48318c0eff17d6c0cec9b362e2ffe975b7e1871a7962dd2c791b1e"}
Oct 30 22:56:38.741534 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:38.741459    4553 scope.go:122] "RemoveContainer" containerID="c0ab0b1cde00524edfd4f0c2ff7826f7952c46290de88120942597dc0e5fc1aa"
Oct 30 22:56:38.751332 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:38.751315    4553 scope.go:122] "RemoveContainer" containerID="c0ab0b1cde00524edfd4f0c2ff7826f7952c46290de88120942597dc0e5fc1aa"
Oct 30 22:56:38.751601 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:56:38.751583    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ab0b1cde00524edfd4f0c2ff7826f7952c46290de88120942597dc0e5fc1aa\": container with ID starting with c0ab0b1cde00524edfd4f0c2ff7826f7952c46290de88120942597dc0e5fc1aa not found: ID does not exist" containerID="c0ab0b1cde00524edfd4f0c2ff7826f7952c46290de88120942597dc0e5fc1aa"
Oct 30 22:56:38.751665 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:38.751609    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ab0b1cde00524edfd4f0c2ff7826f7952c46290de88120942597dc0e5fc1aa"} err="failed to get container status \"c0ab0b1cde00524edfd4f0c2ff7826f7952c46290de88120942597dc0e5fc1aa\": rpc error: code = NotFound desc = could not find container \"c0ab0b1cde00524edfd4f0c2ff7826f7952c46290de88120942597dc0e5fc1aa\": container with ID starting with c0ab0b1cde00524edfd4f0c2ff7826f7952c46290de88120942597dc0e5fc1aa not found: ID does not exist"
Oct 30 22:56:38.908067 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:38.907992    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"]
Oct 30 22:56:38.911522 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:38.911497    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-89cf28-f6847849f-hg28r"]
Oct 30 22:56:39.391921 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:39.391877    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049256e6-63e4-4ae9-9943-5c03d0091451" path="/var/lib/kubelet/pods/049256e6-63e4-4ae9-9943-5c03d0091451/volumes"
Oct 30 22:56:41.550416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:41.550355    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4" podUID="af38adcc-b4d4-4c9b-838e-e2f34115eb95" containerName="payload-processing" containerID="cri-o://017d1e85a430cb12dc5ca6c1bb89ba2fe6ba674fec801911135ce2ec01a28519" gracePeriod=25
Oct 30 22:56:41.659273 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:41.659199    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt" podUID="daf9bde1-9416-46d4-89ed-e81e287a923f" containerName="payload-pre-processing" containerID="cri-o://baff9e43089bfd6de90a888ba7498866a451539032429996a359421fee20c907" gracePeriod=25
Oct 30 22:56:42.238014 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:42.237979    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt" podUID="daf9bde1-9416-46d4-89ed-e81e287a923f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:42.238237 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:42.238010    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4" podUID="af38adcc-b4d4-4c9b-838e-e2f34115eb95" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:43.440366 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.440328    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"]
Oct 30 22:56:43.440816 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.440545    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt" podUID="963f2df0-8b48-4caf-8485-1a8dfe58ee70" containerName="maas-api" containerID="cri-o://5fe39366e8edb51ad8a2c3fe333e09f19c69cadf46f61183da194da851b8915e" gracePeriod=30
Oct 30 22:56:43.703305 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.703236    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:56:43.755019 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.754982    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-metrics-certs\") pod \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") "
Oct 30 22:56:43.755242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.755076    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963f2df0-8b48-4caf-8485-1a8dfe58ee70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963f2df0-8b48-4caf-8485-1a8dfe58ee70-openshift-service-ca\") pod \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") "
Oct 30 22:56:43.755242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.755160    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963f2df0-8b48-4caf-8485-1a8dfe58ee70-kube-api-access-x5wjd\" (UniqueName: \"kubernetes.io/projected/963f2df0-8b48-4caf-8485-1a8dfe58ee70-kube-api-access-x5wjd\") pod \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") "
Oct 30 22:56:43.755242 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.755200    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-maas-api-tls\") pod \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\" (UID: \"963f2df0-8b48-4caf-8485-1a8dfe58ee70\") "
Oct 30 22:56:43.755529 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.755494    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963f2df0-8b48-4caf-8485-1a8dfe58ee70-openshift-service-ca" pod "963f2df0-8b48-4caf-8485-1a8dfe58ee70" (UID: "963f2df0-8b48-4caf-8485-1a8dfe58ee70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:43.757367 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.757346    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-metrics-certs" pod "963f2df0-8b48-4caf-8485-1a8dfe58ee70" (UID: "963f2df0-8b48-4caf-8485-1a8dfe58ee70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:43.757488 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.757464    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963f2df0-8b48-4caf-8485-1a8dfe58ee70-kube-api-access-x5wjd" pod "963f2df0-8b48-4caf-8485-1a8dfe58ee70" (UID: "963f2df0-8b48-4caf-8485-1a8dfe58ee70"). InnerVolumeSpecName "kube-api-access-x5wjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:43.757620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.757599    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-maas-api-tls" pod "963f2df0-8b48-4caf-8485-1a8dfe58ee70" (UID: "963f2df0-8b48-4caf-8485-1a8dfe58ee70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:43.769191 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.769159    4553 generic.go:356] "Generic (PLEG): container finished" podID="963f2df0-8b48-4caf-8485-1a8dfe58ee70" containerID="5fe39366e8edb51ad8a2c3fe333e09f19c69cadf46f61183da194da851b8915e" exitCode=0
Oct 30 22:56:43.769357 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.769202    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"
Oct 30 22:56:43.769357 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.769247    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt" event={"ID":"963f2df0-8b48-4caf-8485-1a8dfe58ee70","Type":"ContainerDied","Data":"5fe39366e8edb51ad8a2c3fe333e09f19c69cadf46f61183da194da851b8915e"}
Oct 30 22:56:43.769357 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.769315    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt" event={"ID":"963f2df0-8b48-4caf-8485-1a8dfe58ee70","Type":"ContainerDied","Data":"4721873145d75cebf080cc37a602fe3a26535545911002dac96053d076722108"}
Oct 30 22:56:43.769596 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.769349    4553 scope.go:122] "RemoveContainer" containerID="5fe39366e8edb51ad8a2c3fe333e09f19c69cadf46f61183da194da851b8915e"
Oct 30 22:56:43.780416 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.780395    4553 scope.go:122] "RemoveContainer" containerID="5fe39366e8edb51ad8a2c3fe333e09f19c69cadf46f61183da194da851b8915e"
Oct 30 22:56:43.780732 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:56:43.780714    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fe39366e8edb51ad8a2c3fe333e09f19c69cadf46f61183da194da851b8915e\": container with ID starting with 5fe39366e8edb51ad8a2c3fe333e09f19c69cadf46f61183da194da851b8915e not found: ID does not exist" containerID="5fe39366e8edb51ad8a2c3fe333e09f19c69cadf46f61183da194da851b8915e"
Oct 30 22:56:43.780789 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.780739    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fe39366e8edb51ad8a2c3fe333e09f19c69cadf46f61183da194da851b8915e"} err="failed to get container status \"5fe39366e8edb51ad8a2c3fe333e09f19c69cadf46f61183da194da851b8915e\": rpc error: code = NotFound desc = could not find container \"5fe39366e8edb51ad8a2c3fe333e09f19c69cadf46f61183da194da851b8915e\": container with ID starting with 5fe39366e8edb51ad8a2c3fe333e09f19c69cadf46f61183da194da851b8915e not found: ID does not exist"
Oct 30 22:56:43.856340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.856306    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963f2df0-8b48-4caf-8485-1a8dfe58ee70-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:43.856340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.856332    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5wjd\" (UniqueName: \"kubernetes.io/projected/963f2df0-8b48-4caf-8485-1a8dfe58ee70-kube-api-access-x5wjd\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:43.856340 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.856345    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:43.856580 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.856355    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963f2df0-8b48-4caf-8485-1a8dfe58ee70-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:56:43.937820 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.937791    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"]
Oct 30 22:56:43.941988 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:43.941964    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt"]
Oct 30 22:56:45.391664 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:45.391631    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963f2df0-8b48-4caf-8485-1a8dfe58ee70" path="/var/lib/kubelet/pods/963f2df0-8b48-4caf-8485-1a8dfe58ee70/volumes"
Oct 30 22:56:46.682042 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:46.681989    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e450a-c7f6db56-g5jlt" podUID="963f2df0-8b48-4caf-8485-1a8dfe58ee70" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.209:8443/health\": context deadline exceeded"
Oct 30 22:56:52.237872 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:52.237831    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt" podUID="daf9bde1-9416-46d4-89ed-e81e287a923f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:52.238284 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:56:52.237831    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4" podUID="af38adcc-b4d4-4c9b-838e-e2f34115eb95" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:58.202303 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:56:58.202264    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:57:01.809951 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:01.809865    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"
Oct 30 22:57:01.860855 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:01.860822    4553 generic.go:356] "Generic (PLEG): container finished" podID="daf9bde1-9416-46d4-89ed-e81e287a923f" containerID="baff9e43089bfd6de90a888ba7498866a451539032429996a359421fee20c907" exitCode=0
Oct 30 22:57:01.861040 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:01.860901    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt" event={"ID":"daf9bde1-9416-46d4-89ed-e81e287a923f","Type":"ContainerDied","Data":"baff9e43089bfd6de90a888ba7498866a451539032429996a359421fee20c907"}
Oct 30 22:57:01.862634 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:01.862601    4553 generic.go:356] "Generic (PLEG): container finished" podID="af38adcc-b4d4-4c9b-838e-e2f34115eb95" containerID="017d1e85a430cb12dc5ca6c1bb89ba2fe6ba674fec801911135ce2ec01a28519" exitCode=0
Oct 30 22:57:01.862824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:01.862663    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"
Oct 30 22:57:01.862824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:01.862680    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4" event={"ID":"af38adcc-b4d4-4c9b-838e-e2f34115eb95","Type":"ContainerDied","Data":"017d1e85a430cb12dc5ca6c1bb89ba2fe6ba674fec801911135ce2ec01a28519"}
Oct 30 22:57:01.862824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:01.862738    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4" event={"ID":"af38adcc-b4d4-4c9b-838e-e2f34115eb95","Type":"ContainerDied","Data":"c336aae85049b736968efdd76a544a8f0675b27595b082994156c9ab993bd503"}
Oct 30 22:57:01.862824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:01.862770    4553 scope.go:122] "RemoveContainer" containerID="017d1e85a430cb12dc5ca6c1bb89ba2fe6ba674fec801911135ce2ec01a28519"
Oct 30 22:57:01.873712 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:01.873691    4553 scope.go:122] "RemoveContainer" containerID="017d1e85a430cb12dc5ca6c1bb89ba2fe6ba674fec801911135ce2ec01a28519"
Oct 30 22:57:01.874032 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:57:01.874003    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017d1e85a430cb12dc5ca6c1bb89ba2fe6ba674fec801911135ce2ec01a28519\": container with ID starting with 017d1e85a430cb12dc5ca6c1bb89ba2fe6ba674fec801911135ce2ec01a28519 not found: ID does not exist" containerID="017d1e85a430cb12dc5ca6c1bb89ba2fe6ba674fec801911135ce2ec01a28519"
Oct 30 22:57:01.874120 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:01.874048    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017d1e85a430cb12dc5ca6c1bb89ba2fe6ba674fec801911135ce2ec01a28519"} err="failed to get container status \"017d1e85a430cb12dc5ca6c1bb89ba2fe6ba674fec801911135ce2ec01a28519\": rpc error: code = NotFound desc = could not find container \"017d1e85a430cb12dc5ca6c1bb89ba2fe6ba674fec801911135ce2ec01a28519\": container with ID starting with 017d1e85a430cb12dc5ca6c1bb89ba2fe6ba674fec801911135ce2ec01a28519 not found: ID does not exist"
Oct 30 22:57:01.911244 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:01.911214    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"
Oct 30 22:57:01.943498 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:01.943470    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af38adcc-b4d4-4c9b-838e-e2f34115eb95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af38adcc-b4d4-4c9b-838e-e2f34115eb95-plugins-config-volume\") pod \"af38adcc-b4d4-4c9b-838e-e2f34115eb95\" (UID: \"af38adcc-b4d4-4c9b-838e-e2f34115eb95\") "
Oct 30 22:57:01.943674 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:01.943572    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af38adcc-b4d4-4c9b-838e-e2f34115eb95-kube-api-access-xvfbw\" (UniqueName: \"kubernetes.io/projected/af38adcc-b4d4-4c9b-838e-e2f34115eb95-kube-api-access-xvfbw\") pod \"af38adcc-b4d4-4c9b-838e-e2f34115eb95\" (UID: \"af38adcc-b4d4-4c9b-838e-e2f34115eb95\") "
Oct 30 22:57:01.943917 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:01.943893    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af38adcc-b4d4-4c9b-838e-e2f34115eb95-plugins-config-volume" pod "af38adcc-b4d4-4c9b-838e-e2f34115eb95" (UID: "af38adcc-b4d4-4c9b-838e-e2f34115eb95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:01.945806 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:01.945782    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af38adcc-b4d4-4c9b-838e-e2f34115eb95-kube-api-access-xvfbw" pod "af38adcc-b4d4-4c9b-838e-e2f34115eb95" (UID: "af38adcc-b4d4-4c9b-838e-e2f34115eb95"). InnerVolumeSpecName "kube-api-access-xvfbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:02.044701 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:02.044671    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf9bde1-9416-46d4-89ed-e81e287a923f-kube-api-access-rpj46\" (UniqueName: \"kubernetes.io/projected/daf9bde1-9416-46d4-89ed-e81e287a923f-kube-api-access-rpj46\") pod \"daf9bde1-9416-46d4-89ed-e81e287a923f\" (UID: \"daf9bde1-9416-46d4-89ed-e81e287a923f\") "
Oct 30 22:57:02.044886 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:02.044797    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daf9bde1-9416-46d4-89ed-e81e287a923f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daf9bde1-9416-46d4-89ed-e81e287a923f-plugins-config-volume\") pod \"daf9bde1-9416-46d4-89ed-e81e287a923f\" (UID: \"daf9bde1-9416-46d4-89ed-e81e287a923f\") "
Oct 30 22:57:02.045011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:02.044998    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvfbw\" (UniqueName: \"kubernetes.io/projected/af38adcc-b4d4-4c9b-838e-e2f34115eb95-kube-api-access-xvfbw\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:02.045054 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:02.045014    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af38adcc-b4d4-4c9b-838e-e2f34115eb95-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:02.045224 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:02.045201    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf9bde1-9416-46d4-89ed-e81e287a923f-plugins-config-volume" pod "daf9bde1-9416-46d4-89ed-e81e287a923f" (UID: "daf9bde1-9416-46d4-89ed-e81e287a923f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:02.047047 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:02.047030    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf9bde1-9416-46d4-89ed-e81e287a923f-kube-api-access-rpj46" pod "daf9bde1-9416-46d4-89ed-e81e287a923f" (UID: "daf9bde1-9416-46d4-89ed-e81e287a923f"). InnerVolumeSpecName "kube-api-access-rpj46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:02.145853 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:02.145813    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpj46\" (UniqueName: \"kubernetes.io/projected/daf9bde1-9416-46d4-89ed-e81e287a923f-kube-api-access-rpj46\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:02.145853 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:02.145847    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daf9bde1-9416-46d4-89ed-e81e287a923f-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:02.285074 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:02.285033    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"]
Oct 30 22:57:02.288697 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:02.288667    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0e450a-76d6769cc9-8q5g4"]
Oct 30 22:57:02.869086 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:02.869053    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"
Oct 30 22:57:02.869629 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:02.869088    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt" event={"ID":"daf9bde1-9416-46d4-89ed-e81e287a923f","Type":"ContainerDied","Data":"2444259b8e26d05f02c39938695b8d26ccab375ec35123301a343e037d94fd32"}
Oct 30 22:57:02.869629 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:02.869186    4553 scope.go:122] "RemoveContainer" containerID="baff9e43089bfd6de90a888ba7498866a451539032429996a359421fee20c907"
Oct 30 22:57:03.037499 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:03.037468    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"]
Oct 30 22:57:03.039546 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:03.039517    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0e450a-6d8db5c787-4pvjt"]
Oct 30 22:57:03.391677 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:03.391642    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af38adcc-b4d4-4c9b-838e-e2f34115eb95" path="/var/lib/kubelet/pods/af38adcc-b4d4-4c9b-838e-e2f34115eb95/volumes"
Oct 30 22:57:03.392048 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:03.392034    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf9bde1-9416-46d4-89ed-e81e287a923f" path="/var/lib/kubelet/pods/daf9bde1-9416-46d4-89ed-e81e287a923f/volumes"
Oct 30 22:57:05.438573 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:05.438537    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"]
Oct 30 22:57:10.439620 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:10.439575    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5" podUID="f11f733e-3c5d-4fb2-b3fa-bbc206983605" containerName="payload-pre-processing" containerID="cri-o://160a078ac404d75ab53630ef4d407e38bcfbcfdf98cc4a84d31e52fb22bf68a4" gracePeriod=25
Oct 30 22:57:16.859505 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:16.859461    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5" podUID="f11f733e-3c5d-4fb2-b3fa-bbc206983605" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:57:25.265746 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.265704    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"]
Oct 30 22:57:25.266236 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.265922    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84" podUID="fb47f460-f5f8-4f9f-908b-4c4532072b3f" containerName="maas-api" containerID="cri-o://e69b643daad0a79160f620d19e185a9124409e57f6fbc1b24bc999e49c85200e" gracePeriod=30
Oct 30 22:57:25.532787 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.532756    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 22:57:25.578291 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.578259    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb47f460-f5f8-4f9f-908b-4c4532072b3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb47f460-f5f8-4f9f-908b-4c4532072b3f-openshift-service-ca\") pod \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") "
Oct 30 22:57:25.578471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.578308    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb47f460-f5f8-4f9f-908b-4c4532072b3f-kube-api-access-2k86k\" (UniqueName: \"kubernetes.io/projected/fb47f460-f5f8-4f9f-908b-4c4532072b3f-kube-api-access-2k86k\") pod \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") "
Oct 30 22:57:25.578471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.578368    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-maas-api-tls\") pod \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") "
Oct 30 22:57:25.578471 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.578436    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-metrics-certs\") pod \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\" (UID: \"fb47f460-f5f8-4f9f-908b-4c4532072b3f\") "
Oct 30 22:57:25.578704 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.578675    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb47f460-f5f8-4f9f-908b-4c4532072b3f-openshift-service-ca" pod "fb47f460-f5f8-4f9f-908b-4c4532072b3f" (UID: "fb47f460-f5f8-4f9f-908b-4c4532072b3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:25.580717 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.580686    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb47f460-f5f8-4f9f-908b-4c4532072b3f-kube-api-access-2k86k" pod "fb47f460-f5f8-4f9f-908b-4c4532072b3f" (UID: "fb47f460-f5f8-4f9f-908b-4c4532072b3f"). InnerVolumeSpecName "kube-api-access-2k86k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:25.580717 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.580703    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-metrics-certs" pod "fb47f460-f5f8-4f9f-908b-4c4532072b3f" (UID: "fb47f460-f5f8-4f9f-908b-4c4532072b3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:25.580921 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.580752    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-maas-api-tls" pod "fb47f460-f5f8-4f9f-908b-4c4532072b3f" (UID: "fb47f460-f5f8-4f9f-908b-4c4532072b3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:57:25.679413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.679371    4553 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-metrics-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:25.679413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.679407    4553 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb47f460-f5f8-4f9f-908b-4c4532072b3f-openshift-service-ca\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:25.679413 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.679419    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k86k\" (UniqueName: \"kubernetes.io/projected/fb47f460-f5f8-4f9f-908b-4c4532072b3f-kube-api-access-2k86k\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:25.679664 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.679428    4553 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb47f460-f5f8-4f9f-908b-4c4532072b3f-maas-api-tls\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:25.979059 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.979025    4553 generic.go:356] "Generic (PLEG): container finished" podID="fb47f460-f5f8-4f9f-908b-4c4532072b3f" containerID="e69b643daad0a79160f620d19e185a9124409e57f6fbc1b24bc999e49c85200e" exitCode=0
Oct 30 22:57:25.979315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.979074    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"
Oct 30 22:57:25.979315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.979110    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84" event={"ID":"fb47f460-f5f8-4f9f-908b-4c4532072b3f","Type":"ContainerDied","Data":"e69b643daad0a79160f620d19e185a9124409e57f6fbc1b24bc999e49c85200e"}
Oct 30 22:57:25.979315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.979205    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84" event={"ID":"fb47f460-f5f8-4f9f-908b-4c4532072b3f","Type":"ContainerDied","Data":"bd8e11746a3de1119599d53d3b5a8c0499f095151bfc260565d57398dfd5648b"}
Oct 30 22:57:25.979315 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.979240    4553 scope.go:122] "RemoveContainer" containerID="e69b643daad0a79160f620d19e185a9124409e57f6fbc1b24bc999e49c85200e"
Oct 30 22:57:25.989368 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.989350    4553 scope.go:122] "RemoveContainer" containerID="e69b643daad0a79160f620d19e185a9124409e57f6fbc1b24bc999e49c85200e"
Oct 30 22:57:25.989677 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:57:25.989658    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69b643daad0a79160f620d19e185a9124409e57f6fbc1b24bc999e49c85200e\": container with ID starting with e69b643daad0a79160f620d19e185a9124409e57f6fbc1b24bc999e49c85200e not found: ID does not exist" containerID="e69b643daad0a79160f620d19e185a9124409e57f6fbc1b24bc999e49c85200e"
Oct 30 22:57:25.989743 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:25.989687    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69b643daad0a79160f620d19e185a9124409e57f6fbc1b24bc999e49c85200e"} err="failed to get container status \"e69b643daad0a79160f620d19e185a9124409e57f6fbc1b24bc999e49c85200e\": rpc error: code = NotFound desc = could not find container \"e69b643daad0a79160f620d19e185a9124409e57f6fbc1b24bc999e49c85200e\": container with ID starting with e69b643daad0a79160f620d19e185a9124409e57f6fbc1b24bc999e49c85200e not found: ID does not exist"
Oct 30 22:57:26.145204 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:26.145167    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"]
Oct 30 22:57:26.150508 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:26.150469    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-e90e35-8ccfd8978-nml84"]
Oct 30 22:57:26.859395 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:26.859362    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5" podUID="f11f733e-3c5d-4fb2-b3fa-bbc206983605" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:57:27.398266 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:27.398226    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb47f460-f5f8-4f9f-908b-4c4532072b3f" path="/var/lib/kubelet/pods/fb47f460-f5f8-4f9f-908b-4c4532072b3f/volumes"
Oct 30 22:57:28.201332 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:57:28.201294    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:57:30.119775 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:30.119745    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:57:30.135770 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:30.135740    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s8pzx_0d0ce829-dc72-4f8e-91ea-2a5313fa7b11/ovn-acl-logging/0.log"
Oct 30 22:57:30.697467 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:30.697405    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"
Oct 30 22:57:30.829447 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:30.829400    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f11f733e-3c5d-4fb2-b3fa-bbc206983605-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f11f733e-3c5d-4fb2-b3fa-bbc206983605-plugins-config-volume\") pod \"f11f733e-3c5d-4fb2-b3fa-bbc206983605\" (UID: \"f11f733e-3c5d-4fb2-b3fa-bbc206983605\") "
Oct 30 22:57:30.829647 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:30.829506    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f11f733e-3c5d-4fb2-b3fa-bbc206983605-kube-api-access-btcfl\" (UniqueName: \"kubernetes.io/projected/f11f733e-3c5d-4fb2-b3fa-bbc206983605-kube-api-access-btcfl\") pod \"f11f733e-3c5d-4fb2-b3fa-bbc206983605\" (UID: \"f11f733e-3c5d-4fb2-b3fa-bbc206983605\") "
Oct 30 22:57:30.829860 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:30.829835    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f11f733e-3c5d-4fb2-b3fa-bbc206983605-plugins-config-volume" pod "f11f733e-3c5d-4fb2-b3fa-bbc206983605" (UID: "f11f733e-3c5d-4fb2-b3fa-bbc206983605"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:30.829956 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:30.829936    4553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f11f733e-3c5d-4fb2-b3fa-bbc206983605-plugins-config-volume\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:30.832022 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:30.831994    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f11f733e-3c5d-4fb2-b3fa-bbc206983605-kube-api-access-btcfl" pod "f11f733e-3c5d-4fb2-b3fa-bbc206983605" (UID: "f11f733e-3c5d-4fb2-b3fa-bbc206983605"). InnerVolumeSpecName "kube-api-access-btcfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:30.931106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:30.931069    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btcfl\" (UniqueName: \"kubernetes.io/projected/f11f733e-3c5d-4fb2-b3fa-bbc206983605-kube-api-access-btcfl\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:31.006564 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:31.006472    4553 generic.go:356] "Generic (PLEG): container finished" podID="f11f733e-3c5d-4fb2-b3fa-bbc206983605" containerID="160a078ac404d75ab53630ef4d407e38bcfbcfdf98cc4a84d31e52fb22bf68a4" exitCode=0
Oct 30 22:57:31.006564 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:31.006526    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"
Oct 30 22:57:31.006769 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:31.006560    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5" event={"ID":"f11f733e-3c5d-4fb2-b3fa-bbc206983605","Type":"ContainerDied","Data":"160a078ac404d75ab53630ef4d407e38bcfbcfdf98cc4a84d31e52fb22bf68a4"}
Oct 30 22:57:31.006769 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:31.006620    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5" event={"ID":"f11f733e-3c5d-4fb2-b3fa-bbc206983605","Type":"ContainerDied","Data":"65c0bc7078c0621a0d9e78551aa6816e9b58181d430b95f077d43293210b8819"}
Oct 30 22:57:31.006769 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:31.006653    4553 scope.go:122] "RemoveContainer" containerID="160a078ac404d75ab53630ef4d407e38bcfbcfdf98cc4a84d31e52fb22bf68a4"
Oct 30 22:57:31.016930 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:31.016909    4553 scope.go:122] "RemoveContainer" containerID="160a078ac404d75ab53630ef4d407e38bcfbcfdf98cc4a84d31e52fb22bf68a4"
Oct 30 22:57:31.017216 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:57:31.017195    4553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"160a078ac404d75ab53630ef4d407e38bcfbcfdf98cc4a84d31e52fb22bf68a4\": container with ID starting with 160a078ac404d75ab53630ef4d407e38bcfbcfdf98cc4a84d31e52fb22bf68a4 not found: ID does not exist" containerID="160a078ac404d75ab53630ef4d407e38bcfbcfdf98cc4a84d31e52fb22bf68a4"
Oct 30 22:57:31.017262 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:31.017229    4553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160a078ac404d75ab53630ef4d407e38bcfbcfdf98cc4a84d31e52fb22bf68a4"} err="failed to get container status \"160a078ac404d75ab53630ef4d407e38bcfbcfdf98cc4a84d31e52fb22bf68a4\": rpc error: code = NotFound desc = could not find container \"160a078ac404d75ab53630ef4d407e38bcfbcfdf98cc4a84d31e52fb22bf68a4\": container with ID starting with 160a078ac404d75ab53630ef4d407e38bcfbcfdf98cc4a84d31e52fb22bf68a4 not found: ID does not exist"
Oct 30 22:57:31.179969 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:31.173446    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"]
Oct 30 22:57:31.183077 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:31.183047    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-e90e35-66b887d98c-tmqg5"]
Oct 30 22:57:31.391816 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:31.391781    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f11f733e-3c5d-4fb2-b3fa-bbc206983605" path="/var/lib/kubelet/pods/f11f733e-3c5d-4fb2-b3fa-bbc206983605/volumes"
Oct 30 22:57:34.837814 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:34.837758    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"]
Oct 30 22:57:34.838435 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:34.838151    4553 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv" podUID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" containerName="istio-proxy" containerID="cri-o://987a0e548c22ea32592598cc7f39bb5588da95c4eb0f4028c33025969e10592d" gracePeriod=30
Oct 30 22:57:38.472461 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:38.472419    4553 patch_prober.go:29] interesting pod/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:57:38.472882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:38.472514    4553 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv" podUID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:57:39.353480 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:39.353444    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6467c7bcbb-q6mdg_cf6aad67-bdb9-4fae-ab13-cf8445563b6c/manager/0.log"
Oct 30 22:57:39.938832 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:39.938787    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fhwfv_63b50514-3b83-44bd-8f3a-6d7d947944c0/manager/0.log"
Oct 30 22:57:40.058576 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.058544    4553 generic.go:356] "Generic (PLEG): container finished" podID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" containerID="987a0e548c22ea32592598cc7f39bb5588da95c4eb0f4028c33025969e10592d" exitCode=0
Oct 30 22:57:40.058774 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.058618    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv" event={"ID":"f8f35f60-8e19-4353-85eb-73b8f7a36c11","Type":"ContainerDied","Data":"987a0e548c22ea32592598cc7f39bb5588da95c4eb0f4028c33025969e10592d"}
Oct 30 22:57:40.058774 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.058698    4553 scope.go:122] "RemoveContainer" containerID="5a94a89a74ee45e53e6272d40cfd852cd87b1b8ec1fd3adf065201de71690ac9"
Oct 30 22:57:40.095478 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.095451    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 22:57:40.222360 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.222259    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istiod-ca-cert\") pod \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") "
Oct 30 22:57:40.222549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.222374    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-workload-certs\") pod \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") "
Oct 30 22:57:40.222549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.222426    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-envoy\") pod \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") "
Oct 30 22:57:40.222549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.222447    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-workload-socket\") pod \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") "
Oct 30 22:57:40.222549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.222469    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-credential-socket\") pod \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") "
Oct 30 22:57:40.222549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.222506    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-podinfo\") pod \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") "
Oct 30 22:57:40.222549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.222548    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f35f60-8e19-4353-85eb-73b8f7a36c11-kube-api-access-vwm8f\" (UniqueName: \"kubernetes.io/projected/f8f35f60-8e19-4353-85eb-73b8f7a36c11-kube-api-access-vwm8f\") pod \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") "
Oct 30 22:57:40.222911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.222635    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-token\" (UniqueName: \"kubernetes.io/projected/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-token\") pod \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") "
Oct 30 22:57:40.222911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.222680    4553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-data\") pod \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\" (UID: \"f8f35f60-8e19-4353-85eb-73b8f7a36c11\") "
Oct 30 22:57:40.222911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.222712    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-workload-certs" pod "f8f35f60-8e19-4353-85eb-73b8f7a36c11" (UID: "f8f35f60-8e19-4353-85eb-73b8f7a36c11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:40.222911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.222791    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-credential-socket" pod "f8f35f60-8e19-4353-85eb-73b8f7a36c11" (UID: "f8f35f60-8e19-4353-85eb-73b8f7a36c11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:40.222911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.222778    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-workload-socket" pod "f8f35f60-8e19-4353-85eb-73b8f7a36c11" (UID: "f8f35f60-8e19-4353-85eb-73b8f7a36c11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:40.222911 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.222811    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istiod-ca-cert" pod "f8f35f60-8e19-4353-85eb-73b8f7a36c11" (UID: "f8f35f60-8e19-4353-85eb-73b8f7a36c11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:57:40.223288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.222998    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-data" pod "f8f35f60-8e19-4353-85eb-73b8f7a36c11" (UID: "f8f35f60-8e19-4353-85eb-73b8f7a36c11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:40.223288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.223082    4553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istiod-ca-cert\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:40.223288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.223109    4553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-workload-certs\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:40.223288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.223124    4553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-workload-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:40.223288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.223174    4553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-credential-socket\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:40.224974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.224947    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-envoy" pod "f8f35f60-8e19-4353-85eb-73b8f7a36c11" (UID: "f8f35f60-8e19-4353-85eb-73b8f7a36c11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:57:40.224974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.224949    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-token" pod "f8f35f60-8e19-4353-85eb-73b8f7a36c11" (UID: "f8f35f60-8e19-4353-85eb-73b8f7a36c11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:40.225186 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.225116    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-podinfo" pod "f8f35f60-8e19-4353-85eb-73b8f7a36c11" (UID: "f8f35f60-8e19-4353-85eb-73b8f7a36c11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:57:40.225186 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.225157    4553 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f35f60-8e19-4353-85eb-73b8f7a36c11-kube-api-access-vwm8f" pod "f8f35f60-8e19-4353-85eb-73b8f7a36c11" (UID: "f8f35f60-8e19-4353-85eb-73b8f7a36c11"). InnerVolumeSpecName "kube-api-access-vwm8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:57:40.324342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.324300    4553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-envoy\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:40.324342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.324337    4553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-podinfo\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:40.324342 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.324346    4553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwm8f\" (UniqueName: \"kubernetes.io/projected/f8f35f60-8e19-4353-85eb-73b8f7a36c11-kube-api-access-vwm8f\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:40.324624 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.324356    4553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-token\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:40.324624 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.324365    4553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8f35f60-8e19-4353-85eb-73b8f7a36c11-istio-data\") on node \"ip-10-0-133-226.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:57:40.783893 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:40.783857    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-655bd8bb49-gwsxw_ab0381a1-70ca-4b49-9e1e-92db62e53c40/manager/0.log"
Oct 30 22:57:41.064663 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:41.064573    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv" event={"ID":"f8f35f60-8e19-4353-85eb-73b8f7a36c11","Type":"ContainerDied","Data":"266fc4f1669c006635dc8c6716a52b206fec1e24f1f8bd7c71a5546d57dcfbf5"}
Oct 30 22:57:41.064663 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:41.064596    4553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"
Oct 30 22:57:41.064663 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:41.064645    4553 scope.go:122] "RemoveContainer" containerID="987a0e548c22ea32592598cc7f39bb5588da95c4eb0f4028c33025969e10592d"
Oct 30 22:57:41.281985 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:41.281939    4553 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"]
Oct 30 22:57:41.332546 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:41.332522    4553 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e90e35-openshift-default-7846d9f897-l8nlv"]
Oct 30 22:57:41.391770 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:57:41.391739    4553 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" path="/var/lib/kubelet/pods/f8f35f60-8e19-4353-85eb-73b8f7a36c11/volumes"
Oct 30 22:57:58.201548 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:57:58.201510    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:58:12.131200 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:12.131066    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6467c7bcbb-q6mdg_cf6aad67-bdb9-4fae-ab13-cf8445563b6c/manager/0.log"
Oct 30 22:58:12.780890 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:12.780847    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fhwfv_63b50514-3b83-44bd-8f3a-6d7d947944c0/manager/0.log"
Oct 30 22:58:13.822659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:13.822626    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-655bd8bb49-gwsxw_ab0381a1-70ca-4b49-9e1e-92db62e53c40/manager/0.log"
Oct 30 22:58:15.365372 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:15.365341    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6c6fd44f6f-p7n4h_af1193f1-5a09-491e-ae7a-3b6ba045097a/maas-api/0.log"
Oct 30 22:58:16.905304 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:16.905268    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-nkpq9_31ee7330-61f5-41ea-967f-23691becb52e/postgres/0.log"
Oct 30 22:58:21.855764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:21.855732    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76f5684988-x9n4b_9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71/kube-auth-proxy/0.log"
Oct 30 22:58:22.513252 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:22.513229    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-2szkr_78f6d7aa-e3b7-45f5-b68a-d6935dda41e6/payload-pre-processing/0.log"
Oct 30 22:58:22.716880 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:22.716792    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-hg9wk_05d7db25-7800-4217-a2fb-4396ba266c73/payload-processing/0.log"
Oct 30 22:58:23.703362 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:23.703310    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-8hvht_8526d260-3d4c-42e8-9bb6-194d1b4fb53d/main/0.log"
Oct 30 22:58:23.965505 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:23.965404    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-5xgvw_deeba91c-c79c-4d40-aea6-ae9e8cc3f4c8/main/0.log"
Oct 30 22:58:24.625395 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:24.625335    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b85cglq_9867d053-6ea1-4062-814a-afdbaa4993f9/main/0.log"
Oct 30 22:58:24.892539 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:24.892440    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-bfhs7_4978a88d-cbce-4aa8-92bc-9f2d8e273651/main/0.log"
Oct 30 22:58:28.201057 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:58:28.201017    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:58:55.480428 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480391    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hkk4m/must-gather-gfz7p"]
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480772    4553 cpu_manager.go:414] "containerMap: removing container" podUID="51d3cb4a-13d1-4198-ab0e-0cf97376e53f" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480783    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d3cb4a-13d1-4198-ab0e-0cf97376e53f" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480791    4553 cpu_manager.go:414] "containerMap: removing container" podUID="c4d6c182-7bf5-4a2c-95d5-174b4f34d298" containerName="maas-api"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480796    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d6c182-7bf5-4a2c-95d5-174b4f34d298" containerName="maas-api"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480807    4553 cpu_manager.go:414] "containerMap: removing container" podUID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480812    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480828    4553 cpu_manager.go:414] "containerMap: removing container" podUID="7103581d-3919-4cb9-bae3-9ade62f0ed1e" containerName="payload-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480833    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7103581d-3919-4cb9-bae3-9ade62f0ed1e" containerName="payload-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480840    4553 cpu_manager.go:414] "containerMap: removing container" podUID="f11f733e-3c5d-4fb2-b3fa-bbc206983605" containerName="payload-pre-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480845    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f11f733e-3c5d-4fb2-b3fa-bbc206983605" containerName="payload-pre-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480850    4553 cpu_manager.go:414] "containerMap: removing container" podUID="62d53659-7ffd-4b08-ae37-8e705017b7d9" containerName="maas-api"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480854    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d53659-7ffd-4b08-ae37-8e705017b7d9" containerName="maas-api"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480860    4553 cpu_manager.go:414] "containerMap: removing container" podUID="12e366ed-b8f7-4d11-aae0-9f3e4015c8cc" containerName="main"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480864    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e366ed-b8f7-4d11-aae0-9f3e4015c8cc" containerName="main"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480877    4553 cpu_manager.go:414] "containerMap: removing container" podUID="c065085c-3963-45a0-b4bc-09ad435e27b7" containerName="payload-pre-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480881    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c065085c-3963-45a0-b4bc-09ad435e27b7" containerName="payload-pre-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480890    4553 cpu_manager.go:414] "containerMap: removing container" podUID="0eaa3e7a-052e-4f6a-9cd3-d3a648528160" containerName="payload-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480894    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eaa3e7a-052e-4f6a-9cd3-d3a648528160" containerName="payload-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480900    4553 cpu_manager.go:414] "containerMap: removing container" podUID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480904    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480909    4553 cpu_manager.go:414] "containerMap: removing container" podUID="af38adcc-b4d4-4c9b-838e-e2f34115eb95" containerName="payload-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480913    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af38adcc-b4d4-4c9b-838e-e2f34115eb95" containerName="payload-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480923    4553 cpu_manager.go:414] "containerMap: removing container" podUID="e2b08777-7389-4a3c-acc8-95e0c5a274bd" containerName="payload-pre-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480929    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b08777-7389-4a3c-acc8-95e0c5a274bd" containerName="payload-pre-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480936    4553 cpu_manager.go:414] "containerMap: removing container" podUID="049256e6-63e4-4ae9-9943-5c03d0091451" containerName="payload-pre-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480940    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049256e6-63e4-4ae9-9943-5c03d0091451" containerName="payload-pre-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480947    4553 cpu_manager.go:414] "containerMap: removing container" podUID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480952    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480958    4553 cpu_manager.go:414] "containerMap: removing container" podUID="daf9bde1-9416-46d4-89ed-e81e287a923f" containerName="payload-pre-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480963    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf9bde1-9416-46d4-89ed-e81e287a923f" containerName="payload-pre-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480973    4553 cpu_manager.go:414] "containerMap: removing container" podUID="d71f867b-5b77-4792-b2d9-71d325c58450" containerName="maas-api"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480978    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71f867b-5b77-4792-b2d9-71d325c58450" containerName="maas-api"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480985    4553 cpu_manager.go:414] "containerMap: removing container" podUID="51d3cb4a-13d1-4198-ab0e-0cf97376e53f" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480989    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d3cb4a-13d1-4198-ab0e-0cf97376e53f" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.480995    4553 cpu_manager.go:414] "containerMap: removing container" podUID="963f2df0-8b48-4caf-8485-1a8dfe58ee70" containerName="maas-api"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481000    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963f2df0-8b48-4caf-8485-1a8dfe58ee70" containerName="maas-api"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481008    4553 cpu_manager.go:414] "containerMap: removing container" podUID="fb47f460-f5f8-4f9f-908b-4c4532072b3f" containerName="maas-api"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481013    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb47f460-f5f8-4f9f-908b-4c4532072b3f" containerName="maas-api"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481078    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="c065085c-3963-45a0-b4bc-09ad435e27b7" containerName="payload-pre-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481087    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eaa3e7a-052e-4f6a-9cd3-d3a648528160" containerName="payload-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481094    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481100    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="af38adcc-b4d4-4c9b-838e-e2f34115eb95" containerName="payload-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481107    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2b08777-7389-4a3c-acc8-95e0c5a274bd" containerName="payload-pre-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481112    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="049256e6-63e4-4ae9-9943-5c03d0091451" containerName="payload-pre-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481117    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f35f60-8e19-4353-85eb-73b8f7a36c11" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481122    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf9bde1-9416-46d4-89ed-e81e287a923f" containerName="payload-pre-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481128    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71f867b-5b77-4792-b2d9-71d325c58450" containerName="maas-api"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481153    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d3cb4a-13d1-4198-ab0e-0cf97376e53f" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481161    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="963f2df0-8b48-4caf-8485-1a8dfe58ee70" containerName="maas-api"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481170    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb47f460-f5f8-4f9f-908b-4c4532072b3f" containerName="maas-api"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481175    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d3cb4a-13d1-4198-ab0e-0cf97376e53f" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481181    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d6c182-7bf5-4a2c-95d5-174b4f34d298" containerName="maas-api"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481186    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481192    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="7103581d-3919-4cb9-bae3-9ade62f0ed1e" containerName="payload-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481198    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="f11f733e-3c5d-4fb2-b3fa-bbc206983605" containerName="payload-pre-processing"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481203    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d53659-7ffd-4b08-ae37-8e705017b7d9" containerName="maas-api"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481207    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e366ed-b8f7-4d11-aae0-9f3e4015c8cc" containerName="main"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481213    4553 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481299    4553 cpu_manager.go:414] "containerMap: removing container" podUID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerName="istio-proxy"
Oct 30 22:58:55.482949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.481307    4553 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d1f1e4c-192c-4eb9-8f74-d9c21d31ad4b" containerName="istio-proxy"
Oct 30 22:58:55.486080 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.485216    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hkk4m/must-gather-gfz7p"
Oct 30 22:58:55.540086 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.540049    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hkk4m/must-gather-gfz7p"]
Oct 30 22:58:55.588766 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.588737    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hkk4m\"/\"openshift-service-ca.crt\""
Oct 30 22:58:55.588998 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.588982    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hkk4m\"/\"kube-root-ca.crt\""
Oct 30 22:58:55.589058 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.588988    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hkk4m\"/\"default-dockercfg-flxp7\""
Oct 30 22:58:55.591240 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.591222    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tnkfv/must-gather-cqr2p"]
Oct 30 22:58:55.594760 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.594743    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-276rx/must-gather-xpnhk"]
Oct 30 22:58:55.594882 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.594864    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tnkfv/must-gather-cqr2p"
Oct 30 22:58:55.598310 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.598291    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-276rx/must-gather-xpnhk"
Oct 30 22:58:55.604469 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.604446    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtr28\" (UniqueName: \"kubernetes.io/projected/33fe9871-8d63-40ca-ba03-ae6dba758bd4-kube-api-access-vtr28\") pod \"must-gather-gfz7p\" (UID: \"33fe9871-8d63-40ca-ba03-ae6dba758bd4\") " pod="openshift-must-gather-hkk4m/must-gather-gfz7p"
Oct 30 22:58:55.604599 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.604497    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/33fe9871-8d63-40ca-ba03-ae6dba758bd4-must-gather-output\") pod \"must-gather-gfz7p\" (UID: \"33fe9871-8d63-40ca-ba03-ae6dba758bd4\") " pod="openshift-must-gather-hkk4m/must-gather-gfz7p"
Oct 30 22:58:55.648615 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.648584    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tnkfv/must-gather-cqr2p"]
Oct 30 22:58:55.650824 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.650765    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tnkfv\"/\"kube-root-ca.crt\""
Oct 30 22:58:55.651068 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.651049    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-276rx\"/\"openshift-service-ca.crt\""
Oct 30 22:58:55.651068 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.651061    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tnkfv\"/\"default-dockercfg-g4g22\""
Oct 30 22:58:55.651262 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.651102    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tnkfv\"/\"openshift-service-ca.crt\""
Oct 30 22:58:55.652659 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.652638    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-276rx/must-gather-xpnhk"]
Oct 30 22:58:55.653815 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.653796    4553 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-276rx\"/\"kube-root-ca.crt\""
Oct 30 22:58:55.653974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.653798    4553 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-276rx\"/\"default-dockercfg-pbpfk\""
Oct 30 22:58:55.705075 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.705048    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqg99\" (UniqueName: \"kubernetes.io/projected/7dc89900-675e-4a00-98ef-2ed7e7fb3fc5-kube-api-access-xqg99\") pod \"must-gather-cqr2p\" (UID: \"7dc89900-675e-4a00-98ef-2ed7e7fb3fc5\") " pod="openshift-must-gather-tnkfv/must-gather-cqr2p"
Oct 30 22:58:55.705293 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.705081    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtr28\" (UniqueName: \"kubernetes.io/projected/33fe9871-8d63-40ca-ba03-ae6dba758bd4-kube-api-access-vtr28\") pod \"must-gather-gfz7p\" (UID: \"33fe9871-8d63-40ca-ba03-ae6dba758bd4\") " pod="openshift-must-gather-hkk4m/must-gather-gfz7p"
Oct 30 22:58:55.705293 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.705118    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/33fe9871-8d63-40ca-ba03-ae6dba758bd4-must-gather-output\") pod \"must-gather-gfz7p\" (UID: \"33fe9871-8d63-40ca-ba03-ae6dba758bd4\") " pod="openshift-must-gather-hkk4m/must-gather-gfz7p"
Oct 30 22:58:55.705293 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.705206    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/765dc272-c82d-4484-b115-8e5dccdd237b-must-gather-output\") pod \"must-gather-xpnhk\" (UID: \"765dc272-c82d-4484-b115-8e5dccdd237b\") " pod="openshift-must-gather-276rx/must-gather-xpnhk"
Oct 30 22:58:55.705293 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.705242    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv2zm\" (UniqueName: \"kubernetes.io/projected/765dc272-c82d-4484-b115-8e5dccdd237b-kube-api-access-bv2zm\") pod \"must-gather-xpnhk\" (UID: \"765dc272-c82d-4484-b115-8e5dccdd237b\") " pod="openshift-must-gather-276rx/must-gather-xpnhk"
Oct 30 22:58:55.705293 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.705277    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7dc89900-675e-4a00-98ef-2ed7e7fb3fc5-must-gather-output\") pod \"must-gather-cqr2p\" (UID: \"7dc89900-675e-4a00-98ef-2ed7e7fb3fc5\") " pod="openshift-must-gather-tnkfv/must-gather-cqr2p"
Oct 30 22:58:55.705556 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.705422    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/33fe9871-8d63-40ca-ba03-ae6dba758bd4-must-gather-output\") pod \"must-gather-gfz7p\" (UID: \"33fe9871-8d63-40ca-ba03-ae6dba758bd4\") " pod="openshift-must-gather-hkk4m/must-gather-gfz7p"
Oct 30 22:58:55.763011 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.762934    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtr28\" (UniqueName: \"kubernetes.io/projected/33fe9871-8d63-40ca-ba03-ae6dba758bd4-kube-api-access-vtr28\") pod \"must-gather-gfz7p\" (UID: \"33fe9871-8d63-40ca-ba03-ae6dba758bd4\") " pod="openshift-must-gather-hkk4m/must-gather-gfz7p"
Oct 30 22:58:55.801598 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.801560    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hkk4m/must-gather-gfz7p"
Oct 30 22:58:55.806090 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.806057    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2zm\" (UniqueName: \"kubernetes.io/projected/765dc272-c82d-4484-b115-8e5dccdd237b-kube-api-access-bv2zm\") pod \"must-gather-xpnhk\" (UID: \"765dc272-c82d-4484-b115-8e5dccdd237b\") " pod="openshift-must-gather-276rx/must-gather-xpnhk"
Oct 30 22:58:55.806233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.806157    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7dc89900-675e-4a00-98ef-2ed7e7fb3fc5-must-gather-output\") pod \"must-gather-cqr2p\" (UID: \"7dc89900-675e-4a00-98ef-2ed7e7fb3fc5\") " pod="openshift-must-gather-tnkfv/must-gather-cqr2p"
Oct 30 22:58:55.806283 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.806232    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqg99\" (UniqueName: \"kubernetes.io/projected/7dc89900-675e-4a00-98ef-2ed7e7fb3fc5-kube-api-access-xqg99\") pod \"must-gather-cqr2p\" (UID: \"7dc89900-675e-4a00-98ef-2ed7e7fb3fc5\") " pod="openshift-must-gather-tnkfv/must-gather-cqr2p"
Oct 30 22:58:55.806346 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.806325    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/765dc272-c82d-4484-b115-8e5dccdd237b-must-gather-output\") pod \"must-gather-xpnhk\" (UID: \"765dc272-c82d-4484-b115-8e5dccdd237b\") " pod="openshift-must-gather-276rx/must-gather-xpnhk"
Oct 30 22:58:55.806549 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.806528    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7dc89900-675e-4a00-98ef-2ed7e7fb3fc5-must-gather-output\") pod \"must-gather-cqr2p\" (UID: \"7dc89900-675e-4a00-98ef-2ed7e7fb3fc5\") " pod="openshift-must-gather-tnkfv/must-gather-cqr2p"
Oct 30 22:58:55.806694 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.806674    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/765dc272-c82d-4484-b115-8e5dccdd237b-must-gather-output\") pod \"must-gather-xpnhk\" (UID: \"765dc272-c82d-4484-b115-8e5dccdd237b\") " pod="openshift-must-gather-276rx/must-gather-xpnhk"
Oct 30 22:58:55.870160 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.870104    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv2zm\" (UniqueName: \"kubernetes.io/projected/765dc272-c82d-4484-b115-8e5dccdd237b-kube-api-access-bv2zm\") pod \"must-gather-xpnhk\" (UID: \"765dc272-c82d-4484-b115-8e5dccdd237b\") " pod="openshift-must-gather-276rx/must-gather-xpnhk"
Oct 30 22:58:55.873171 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.872882    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqg99\" (UniqueName: \"kubernetes.io/projected/7dc89900-675e-4a00-98ef-2ed7e7fb3fc5-kube-api-access-xqg99\") pod \"must-gather-cqr2p\" (UID: \"7dc89900-675e-4a00-98ef-2ed7e7fb3fc5\") " pod="openshift-must-gather-tnkfv/must-gather-cqr2p"
Oct 30 22:58:55.911178 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.911110    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tnkfv/must-gather-cqr2p"
Oct 30 22:58:55.916252 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:55.916236    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-276rx/must-gather-xpnhk"
Oct 30 22:58:56.177298 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:56.177272    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hkk4m/must-gather-gfz7p"]
Oct 30 22:58:56.181531 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:56.181508    4553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:58:56.297354 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:56.297328    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tnkfv/must-gather-cqr2p"]
Oct 30 22:58:56.299438 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:58:56.299413    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7dc89900_675e_4a00_98ef_2ed7e7fb3fc5.slice/crio-b494e187483f531bf8eca00b6de8012761f71849278c19b6c5d5d44d5c0b7a1f WatchSource:0}: Error finding container b494e187483f531bf8eca00b6de8012761f71849278c19b6c5d5d44d5c0b7a1f: Status 404 returned error can't find the container with id b494e187483f531bf8eca00b6de8012761f71849278c19b6c5d5d44d5c0b7a1f
Oct 30 22:58:56.346452 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:56.346429    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-276rx/must-gather-xpnhk"]
Oct 30 22:58:56.347940 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:58:56.347917    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod765dc272_c82d_4484_b115_8e5dccdd237b.slice/crio-c923c9f184ffa4b9146e70d0968c0174f680b00321280a8ae8a6a44fbeb807f1 WatchSource:0}: Error finding container c923c9f184ffa4b9146e70d0968c0174f680b00321280a8ae8a6a44fbeb807f1: Status 404 returned error can't find the container with id c923c9f184ffa4b9146e70d0968c0174f680b00321280a8ae8a6a44fbeb807f1
Oct 30 22:58:56.405385 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:56.405338    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hkk4m/must-gather-gfz7p" event={"ID":"33fe9871-8d63-40ca-ba03-ae6dba758bd4","Type":"ContainerStarted","Data":"9b3ac234c697865e0cecbbb4faeb47a535cbb195ed9d5f6cf7a0f250619e4feb"}
Oct 30 22:58:56.406523 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:56.406489    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-276rx/must-gather-xpnhk" event={"ID":"765dc272-c82d-4484-b115-8e5dccdd237b","Type":"ContainerStarted","Data":"c923c9f184ffa4b9146e70d0968c0174f680b00321280a8ae8a6a44fbeb807f1"}
Oct 30 22:58:56.407645 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:56.407619    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tnkfv/must-gather-cqr2p" event={"ID":"7dc89900-675e-4a00-98ef-2ed7e7fb3fc5","Type":"ContainerStarted","Data":"b494e187483f531bf8eca00b6de8012761f71849278c19b6c5d5d44d5c0b7a1f"}
Oct 30 22:58:58.210236 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:58:58.210196    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:58:58.419006 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:58.418967    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-276rx/must-gather-xpnhk" event={"ID":"765dc272-c82d-4484-b115-8e5dccdd237b","Type":"ContainerStarted","Data":"207bab2009ebbff67ab9df7516a248b114b5d74b0e9795004b8e257988bde9ca"}
Oct 30 22:58:58.419232 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:58.419015    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-276rx/must-gather-xpnhk" event={"ID":"765dc272-c82d-4484-b115-8e5dccdd237b","Type":"ContainerStarted","Data":"a16aa8d50035eb8710f17c85c19990f665f43e21247aed7063c7f8a281e805e4"}
Oct 30 22:58:58.551949 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:58:58.551813    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-276rx/must-gather-xpnhk" podStartSLOduration=2.305271409 podStartE2EDuration="3.551794116s" totalImagesPullingTime="1.246522707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:58:58.551023776 +0000 UTC m=+4589.285386808" watchObservedRunningTime="2026-10-30 22:58:58.551794116 +0000 UTC m=+4589.286157145"
Oct 30 22:59:05.481385 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:05.481348    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hkk4m/must-gather-gfz7p" event={"ID":"33fe9871-8d63-40ca-ba03-ae6dba758bd4","Type":"ContainerStarted","Data":"8211d40ebecb96c236da782364b5d3b27f65c0c2ce5c6f6304180b9807533d53"}
Oct 30 22:59:06.164157 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:06.162383    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x7hcg_4d1fccd5-6eaa-4a43-9a1e-d67c1005e369/global-pull-secret-syncer/0.log"
Oct 30 22:59:06.515220 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:06.514003    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hkk4m/must-gather-gfz7p" event={"ID":"33fe9871-8d63-40ca-ba03-ae6dba758bd4","Type":"ContainerStarted","Data":"a74721701859df94c550572b9cba068f14ad1b76dd6df51d1fda76d9c512a2af"}
Oct 30 22:59:06.528159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:06.517123    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tnkfv/must-gather-cqr2p" event={"ID":"7dc89900-675e-4a00-98ef-2ed7e7fb3fc5","Type":"ContainerStarted","Data":"e70f4889b073fe46a769a803682e7e8f2914305d9f30ad8dfc4faf54b13c874a"}
Oct 30 22:59:06.528159 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:06.525227    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tnkfv/must-gather-cqr2p" event={"ID":"7dc89900-675e-4a00-98ef-2ed7e7fb3fc5","Type":"ContainerStarted","Data":"169aba66328e0f067d0d5784b8da132ed978bbc0f10547f7046ad9de65ee59cb"}
Oct 30 22:59:06.633218 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:06.632515    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hkk4m/must-gather-gfz7p" podStartSLOduration=2.516464033 podStartE2EDuration="11.632493069s" totalImagesPullingTime="9.116029036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:06.630575379 +0000 UTC m=+4597.364938413" watchObservedRunningTime="2026-10-30 22:59:06.632493069 +0000 UTC m=+4597.366856101"
Oct 30 22:59:06.755607 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:06.754957    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-tnkfv/must-gather-cqr2p" podStartSLOduration=2.699217228 podStartE2EDuration="11.754940238s" totalImagesPullingTime="9.05572301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:06.753054434 +0000 UTC m=+4597.487417466" watchObservedRunningTime="2026-10-30 22:59:06.754940238 +0000 UTC m=+4597.489303266"
Oct 30 22:59:09.611166 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:09.610373    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-prfrp_88edf321-028a-4bb5-b729-1359bd54fb57/konnectivity-agent/0.log"
Oct 30 22:59:10.548158 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:10.547900    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-226.us-west-1.compute.internal_24d6104f1ec87a54e6dc2f912238418d/haproxy/0.log"
Oct 30 22:59:11.556906 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:11.556868    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6467c7bcbb-q6mdg_cf6aad67-bdb9-4fae-ab13-cf8445563b6c/manager/0.log"
Oct 30 22:59:11.713959 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:11.713926    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6467c7bcbb-q6mdg_cf6aad67-bdb9-4fae-ab13-cf8445563b6c/manager/0.log"
Oct 30 22:59:12.014325 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:12.014292    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fhwfv_63b50514-3b83-44bd-8f3a-6d7d947944c0/manager/0.log"
Oct 30 22:59:12.175690 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:12.175660    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fhwfv_63b50514-3b83-44bd-8f3a-6d7d947944c0/manager/0.log"
Oct 30 22:59:13.230231 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:13.229704    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-655bd8bb49-gwsxw_ab0381a1-70ca-4b49-9e1e-92db62e53c40/manager/0.log"
Oct 30 22:59:13.481204 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:13.480304    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-655bd8bb49-gwsxw_ab0381a1-70ca-4b49-9e1e-92db62e53c40/manager/0.log"
Oct 30 22:59:17.280463 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.280425    4553 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"]
Oct 30 22:59:17.289502 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.289474    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.305254 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.305219    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crdb8\" (UniqueName: \"kubernetes.io/projected/05aa514c-5ff9-4ae1-947f-3710a784e242-kube-api-access-crdb8\") pod \"perf-node-gather-daemonset-cln9k\" (UID: \"05aa514c-5ff9-4ae1-947f-3710a784e242\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.305438 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.305299    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/05aa514c-5ff9-4ae1-947f-3710a784e242-podres\") pod \"perf-node-gather-daemonset-cln9k\" (UID: \"05aa514c-5ff9-4ae1-947f-3710a784e242\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.305504 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.305492    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05aa514c-5ff9-4ae1-947f-3710a784e242-lib-modules\") pod \"perf-node-gather-daemonset-cln9k\" (UID: \"05aa514c-5ff9-4ae1-947f-3710a784e242\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.305634 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.305616    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05aa514c-5ff9-4ae1-947f-3710a784e242-sys\") pod \"perf-node-gather-daemonset-cln9k\" (UID: \"05aa514c-5ff9-4ae1-947f-3710a784e242\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.305693 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.305666    4553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/05aa514c-5ff9-4ae1-947f-3710a784e242-proc\") pod \"perf-node-gather-daemonset-cln9k\" (UID: \"05aa514c-5ff9-4ae1-947f-3710a784e242\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.367233 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.367201    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"]
Oct 30 22:59:17.406361 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.406318    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05aa514c-5ff9-4ae1-947f-3710a784e242-sys\") pod \"perf-node-gather-daemonset-cln9k\" (UID: \"05aa514c-5ff9-4ae1-947f-3710a784e242\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.406361 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.406353    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/05aa514c-5ff9-4ae1-947f-3710a784e242-proc\") pod \"perf-node-gather-daemonset-cln9k\" (UID: \"05aa514c-5ff9-4ae1-947f-3710a784e242\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.406593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.406398    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crdb8\" (UniqueName: \"kubernetes.io/projected/05aa514c-5ff9-4ae1-947f-3710a784e242-kube-api-access-crdb8\") pod \"perf-node-gather-daemonset-cln9k\" (UID: \"05aa514c-5ff9-4ae1-947f-3710a784e242\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.406593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.406429    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/05aa514c-5ff9-4ae1-947f-3710a784e242-podres\") pod \"perf-node-gather-daemonset-cln9k\" (UID: \"05aa514c-5ff9-4ae1-947f-3710a784e242\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.406593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.406437    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05aa514c-5ff9-4ae1-947f-3710a784e242-sys\") pod \"perf-node-gather-daemonset-cln9k\" (UID: \"05aa514c-5ff9-4ae1-947f-3710a784e242\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.406593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.406450    4553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05aa514c-5ff9-4ae1-947f-3710a784e242-lib-modules\") pod \"perf-node-gather-daemonset-cln9k\" (UID: \"05aa514c-5ff9-4ae1-947f-3710a784e242\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.406593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.406520    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/05aa514c-5ff9-4ae1-947f-3710a784e242-podres\") pod \"perf-node-gather-daemonset-cln9k\" (UID: \"05aa514c-5ff9-4ae1-947f-3710a784e242\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.406593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.406553    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/05aa514c-5ff9-4ae1-947f-3710a784e242-proc\") pod \"perf-node-gather-daemonset-cln9k\" (UID: \"05aa514c-5ff9-4ae1-947f-3710a784e242\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.406593 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.406582    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05aa514c-5ff9-4ae1-947f-3710a784e242-lib-modules\") pod \"perf-node-gather-daemonset-cln9k\" (UID: \"05aa514c-5ff9-4ae1-947f-3710a784e242\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.466113 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.466081    4553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crdb8\" (UniqueName: \"kubernetes.io/projected/05aa514c-5ff9-4ae1-947f-3710a784e242-kube-api-access-crdb8\") pod \"perf-node-gather-daemonset-cln9k\" (UID: \"05aa514c-5ff9-4ae1-947f-3710a784e242\") " pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:17.603008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:17.602970    4553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:18.003106 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:18.003077    4553 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"]
Oct 30 22:59:18.006412 ip-10-0-133-226 kubenswrapper[4553]: W1030 22:59:18.006365    4553 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod05aa514c_5ff9_4ae1_947f_3710a784e242.slice/crio-ce4827825271a88435a1d6f3ba3d3a5fe17cef8ae957300a960e8438afddd2ce WatchSource:0}: Error finding container ce4827825271a88435a1d6f3ba3d3a5fe17cef8ae957300a960e8438afddd2ce: Status 404 returned error can't find the container with id ce4827825271a88435a1d6f3ba3d3a5fe17cef8ae957300a960e8438afddd2ce
Oct 30 22:59:18.432646 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:18.432620    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6467c7bcbb-q6mdg_cf6aad67-bdb9-4fae-ab13-cf8445563b6c/manager/0.log"
Oct 30 22:59:18.477538 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:18.477506    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6467c7bcbb-q6mdg_cf6aad67-bdb9-4fae-ab13-cf8445563b6c/manager/0.log"
Oct 30 22:59:18.656990 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:18.656948    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k" event={"ID":"05aa514c-5ff9-4ae1-947f-3710a784e242","Type":"ContainerStarted","Data":"20707f714ad6174638863078c93b86b85ae7f707a620f0c66995dc804922ef0a"}
Oct 30 22:59:18.657237 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:18.657016    4553 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k" event={"ID":"05aa514c-5ff9-4ae1-947f-3710a784e242","Type":"ContainerStarted","Data":"ce4827825271a88435a1d6f3ba3d3a5fe17cef8ae957300a960e8438afddd2ce"}
Oct 30 22:59:18.657237 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:18.657186    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:18.775483 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:18.775375    4553 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k" podStartSLOduration=1.7753591420000001 podStartE2EDuration="1.775359142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:18.773826423 +0000 UTC m=+4609.508189455" watchObservedRunningTime="2026-10-30 22:59:18.775359142 +0000 UTC m=+4609.509722172"
Oct 30 22:59:18.893600 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:18.893532    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fhwfv_63b50514-3b83-44bd-8f3a-6d7d947944c0/manager/0.log"
Oct 30 22:59:18.947764 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:18.947709    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fhwfv_63b50514-3b83-44bd-8f3a-6d7d947944c0/manager/0.log"
Oct 30 22:59:20.333674 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:20.333633    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-655bd8bb49-gwsxw_ab0381a1-70ca-4b49-9e1e-92db62e53c40/manager/0.log"
Oct 30 22:59:20.417664 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:20.417604    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-655bd8bb49-gwsxw_ab0381a1-70ca-4b49-9e1e-92db62e53c40/manager/0.log"
Oct 30 22:59:24.674081 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:24.674045    4553 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-276rx/perf-node-gather-daemonset-cln9k"
Oct 30 22:59:25.239770 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:25.239724    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6467c7bcbb-q6mdg_cf6aad67-bdb9-4fae-ab13-cf8445563b6c/manager/0.log"
Oct 30 22:59:25.297398 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:25.297356    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6467c7bcbb-q6mdg_cf6aad67-bdb9-4fae-ab13-cf8445563b6c/manager/0.log"
Oct 30 22:59:25.433407 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:25.433369    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vjssf_62254772-614a-4bba-a726-6866cd578123/kube-state-metrics/0.log"
Oct 30 22:59:25.680528 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:25.680496    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fhwfv_63b50514-3b83-44bd-8f3a-6d7d947944c0/manager/0.log"
Oct 30 22:59:25.785669 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:25.785639    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-fhwfv_63b50514-3b83-44bd-8f3a-6d7d947944c0/manager/0.log"
Oct 30 22:59:25.849649 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:25.849604    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vjssf_62254772-614a-4bba-a726-6866cd578123/kube-rbac-proxy-main/0.log"
Oct 30 22:59:26.271206 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:26.271175    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-vjssf_62254772-614a-4bba-a726-6866cd578123/kube-rbac-proxy-self/0.log"
Oct 30 22:59:26.695484 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:26.695451    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-848d46785f-km9rb_47537107-9197-4436-b13e-d675c5655e04/metrics-server/0.log"
Oct 30 22:59:26.992114 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:26.992031    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-655bd8bb49-gwsxw_ab0381a1-70ca-4b49-9e1e-92db62e53c40/manager/0.log"
Oct 30 22:59:27.121190 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:27.121127    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-655bd8bb49-gwsxw_ab0381a1-70ca-4b49-9e1e-92db62e53c40/manager/0.log"
Oct 30 22:59:28.208776 ip-10-0-133-226 kubenswrapper[4553]: E1030 22:59:28.208738    4553 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nx7wb" containerName="extractor"
Oct 30 22:59:30.153531 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:30.153491    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sbg2m_8ffe6dba-999b-4dec-ac96-20124d51b0f7/node-exporter/0.log"
Oct 30 22:59:30.574642 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:30.574571    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sbg2m_8ffe6dba-999b-4dec-ac96-20124d51b0f7/kube-rbac-proxy/0.log"
Oct 30 22:59:30.995356 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:30.995326    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sbg2m_8ffe6dba-999b-4dec-ac96-20124d51b0f7/init-textfile/0.log"
Oct 30 22:59:31.430691 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:31.430660    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rcbl7_c58cf9e5-68a2-4fd7-a364-d7a82686ee5d/kube-rbac-proxy-main/0.log"
Oct 30 22:59:31.852876 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:31.852832    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rcbl7_c58cf9e5-68a2-4fd7-a364-d7a82686ee5d/kube-rbac-proxy-self/0.log"
Oct 30 22:59:32.274845 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:32.274762    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-rcbl7_c58cf9e5-68a2-4fd7-a364-d7a82686ee5d/openshift-state-metrics/0.log"
Oct 30 22:59:35.435309 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:35.435274    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76f5684988-x9n4b_9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71/kube-auth-proxy/0.log"
Oct 30 22:59:35.661317 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:35.661291    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76f5684988-x9n4b_9564ef8e-5d55-4ece-a63f-d5d3b6bc3a71/kube-auth-proxy/0.log"
Oct 30 22:59:35.731386 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:35.731311    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-kdqlc_4e62e696-0ac6-43ec-bdf5-d24edc3e06d3/prometheus-operator/0.log"
Oct 30 22:59:36.079008 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:36.078928    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-2szkr_78f6d7aa-e3b7-45f5-b68a-d6935dda41e6/payload-pre-processing/0.log"
Oct 30 22:59:36.148993 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:36.148962    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-kdqlc_4e62e696-0ac6-43ec-bdf5-d24edc3e06d3/kube-rbac-proxy/0.log"
Oct 30 22:59:36.291725 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:36.291688    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-hg9wk_05d7db25-7800-4217-a2fb-4396ba266c73/payload-processing/0.log"
Oct 30 22:59:36.303785 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:36.303733    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-2szkr_78f6d7aa-e3b7-45f5-b68a-d6935dda41e6/payload-pre-processing/0.log"
Oct 30 22:59:36.518727 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:36.518691    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-hg9wk_05d7db25-7800-4217-a2fb-4396ba266c73/payload-processing/0.log"
Oct 30 22:59:38.261407 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:38.261380    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb75f44d8-rclhz_65112dc9-5d05-4aa9-abcc-4eed6b6fb882/thanos-query/0.log"
Oct 30 22:59:38.683537 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:38.683507    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb75f44d8-rclhz_65112dc9-5d05-4aa9-abcc-4eed6b6fb882/kube-rbac-proxy-web/0.log"
Oct 30 22:59:39.117358 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:39.117331    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb75f44d8-rclhz_65112dc9-5d05-4aa9-abcc-4eed6b6fb882/kube-rbac-proxy/0.log"
Oct 30 22:59:39.547696 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:39.547618    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb75f44d8-rclhz_65112dc9-5d05-4aa9-abcc-4eed6b6fb882/prom-label-proxy/0.log"
Oct 30 22:59:39.969974 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:39.969943    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb75f44d8-rclhz_65112dc9-5d05-4aa9-abcc-4eed6b6fb882/kube-rbac-proxy-rules/0.log"
Oct 30 22:59:40.390288 ip-10-0-133-226 kubenswrapper[4553]: I1030 22:59:40.390258    4553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bb75f44d8-rclhz_65112dc9-5d05-4aa9-abcc-4eed6b6fb882/kube-rbac-proxy-metrics/0.log"